Showing posts with label testing. Show all posts
Showing posts with label testing. Show all posts

Friday, 22 September 2017

Testing Functions Locally with the Cloud Functions Shell

Doug Stevenson
Doug Stevenson
Developer Advocate
If you've been working with Cloud Functions for Firebase, you've probably wondered how you could speed up the development of your functions. This is possible for HTTPS type functions using the firebase serve in the Firebase CLI, but this wasn't an option for other types of functions. Now, local testing of all of your functions is easy with the Firebase CLI. If you want to try out your code before you deploy it to Cloud Functions, you can do that with the Cloud Functions shell in the Firebase CLI starting at version 3.11.0 or later.

Here's how it works, in a nutshell. We'll use a Realtime Database trigger as an example.

Imagine you have an existing project with a single function in it called makeUppercase. It doesn't have to be deployed yet, just defined in your index.js:


exports.makeUppercase = functions.database.ref('/messages/{pushId}/original').onCreate(event => {
const original = event.data.val()
console.log('Uppercasing', event.params.pushId, original)
const uppercase = original.toUpperCase()
return event.data.ref.parent.child('uppercase').set(uppercase)
})

This onCreate database trigger runs when a new message is pushed under /messages with a child called original, and writes back to that message a new child called uppercased with the original value capitalized.

Now, if you can kick off the emulator shell from your command line using the Firebase CLI:


$ cd your_project_dir
$ firebase experimental:functions:shell

Then, you'll see something like this:


i functions: Preparing to emulate functions.
✔ functions: makeUppercase
firebase>

That firebase prompt is waiting there for you to issue some commands to invoke your makeUppercase function. The documentation for testing database triggers says that you can use the following syntax to invoke the function with incoming data to describe the event:


makeUppercase('foo')

This emulates the trigger of an event that would be generated when a new message object is pushed under /messages that has a child named original with the string value "foo". When you run this command in the shell, it will generate some output at the console like this:


info: User function triggered, starting execution
info: Uppercasing pushId1 foo
info: Execution took 892 ms, user function completed successfully

Notice that the console log in the function is printed, and it shows that the database path wildcard pushId was automatically assigned the value pushId1 for you. Very convenient! But you can still specify the wildcard values yourself, if you prefer:


makeUppercase('foo', {params: {pushId: 'custom_push_id'}})

After emulating this function, if you look inside the database, you should also see the results of the function on display, with /messages/{pushId}/uppercased set to the uppercased string string value "FOO".

You can simulate any database event this way (onCreate, onDelete, onUpdate, onWrite). Be sure to read the docs to learn how to invoke them each correctly.

In addition to database triggers, you can also emulate HTTPS functions, PubSub functions, Analytics functions, Storage functions, and Auth functions, each with their own special syntax.

The Cloud Functions shell is currently an experimental offering, and as such, you may experience some rough edges. If you encounter a problem, please let us know by filing a bug report. You can also talk to other Cloud Functions users on the Firebase Slack in the #functions channel.

Some tips for using the shell

Typing the function invocation each time can be kind of a pain, so be sure to take advantage of the fact that you can navigate and repurpose your invocation history much like you would your shell's command line using the arrow keys.

Also note that the shell is actually a full node REPL that you can use to execute arbitrary JavaScript code and use special REPL commands and keys. This can be useful for scripting some of your test code.

Since you can execute arbitrary code, you can also dynamically load and execute code from other files using the require() function that you're probably already familiar with.

And lastly, if you're like me, and you prefer to use a programmer's editor such as VS Code to write your all JavaScript, you can easily emulate functions by sending code you want to run to the Firebase CLI. This command will run test code from a file redirected through standard input:


$ firebase experimental:functions:shell < tests.js

Happy testing!

Share:

Thursday, 27 July 2017

Find More Bugs Using StrictMode with Firebase Test Lab for Android

Doug Stevenson
Doug Stevenson
Developer Advocate

Sometimes the worst bugs to track down are the ones that seem to be impossible to reproduce. Or worse, inconsistent performance problems that can cause "Application Not Responding" errors in Android apps. No matter how much test code you write, these types of errors seem to have a way of sneaking into your app and causing problems for your users. However, with some clever use of Android's StrictModeAPI, alongside Firebase Test Lab for Android, you can find out about these problems before they reach production!

Over the years since Android was first available, a number of best practices have been observed for writing solid apps. For example, you shouldn't be performing blocking I/O on the main thread, and you shouldn't store references to Activity objects that are held after the Activity is destroyed. While it's likely that no one will force you to observe these practices (and you may never see a problem during development), enabling StrictMode in your app will let you know where you've made a mistake. These are called "policy violations", and you configure these policies with code in your app.

Personally, I don't want any of my code to cause any StrictMode violations. In fact, I'd like to consider them bugs that are just as serious as a regular crash, because they could crash my app in some cases! And, in fact, I can configure all StrictMode violations to crash the app. The Java code for that looks like this:

    
private static final StrictMode.ThreadPolicy FATAL_THREAD_POLICY =
new StrictMode.ThreadPolicy.Builder()
.detectAll()
.penaltyLog()
.penaltyDeath()
.build();

private static final StrictMode.VmPolicy FATAL_VM_POLICY =
new StrictMode.VmPolicy.Builder()
.detectAll()
.penaltyLog()
.penaltyDeath()
.build();

public static void enableFatalStrictMode() {
if (shouldEnableFatalStrictMode()) {
StrictMode.setThreadPolicy(FATAL_THREAD_POLICY);
StrictMode.setVmPolicy(FATAL_VM_POLICY);
}
}

Here I have both a ThreadPolicy(for the main thread) and a VmPolicy(for the entire app) that look for all known violations, and will both log that violation and crash the app. Take a look through the javadoc for those builders to learn about all the situations they can catch.

Also notice that I'm enabling the policies conditionally. You probably don't want an accidental violation to crash on your users in production, so the method shouldEnableFatalStrictMode() typically looks like this, which activates it for your debug builds only:

    
private static boolean shouldEnableFatalStrictMode() {
return BuildConfig.DEBUG;
}

What I'd like to do is take this further and have StrictMode crash my app also when it's running in Firebase Test Lab. This is handy because the crash will fail the test, and I'll get a report about that in the test results. For that to happen, I can change the method to query a system setting on the device that's unique to devices in Test Lab. Note that this requires an Android Context to obtain a ContentResolver:

    
private static boolean shouldEnableFatalStrictMode() {
ContentResolver resolver = context.getContentResolver();
String isInTestLab = Settings.System.getString(resolver, "firebase.test.lab");
return BuildConfig.DEBUG || "true".equals(isInTestLab);
}

Now, my instrumented tests (and the automated Robo test) will crash with any StrictMode violations, and I'll see those very clearly in my Test Lab report. Here's an app that crashed in Test Lab with a StrictMode violation during a Robo test:

This doesn't tell you exactly what happened, though. Digging into the logs in the test report, I see the details of the violation right before the crash:


D/StrictMode(6996): StrictMode policy violation; ~duration=80 ms: android.os.StrictMode$StrictModeDiskReadViolation: policy=327743 violation=2
D/StrictMode(6996): at android.os.StrictMode$AndroidBlockGuardPolicy.onReadFromDisk(StrictMode.java:1415)
D/StrictMode(6996): at java.io.UnixFileSystem.checkAccess(UnixFileSystem.java:251)
D/StrictMode(6996): at java.io.File.exists(File.java:807)
D/StrictMode(6996): at android.app.ContextImpl.getDataDir(ContextImpl.java:2167)
D/StrictMode(6996): at android.app.ContextImpl.getPreferencesDir(ContextImpl.java:498)
D/StrictMode(6996): at android.app.ContextImpl.getSharedPreferencesPath(ContextImpl.java:692)
D/StrictMode(6996): at android.app.ContextImpl.getSharedPreferences(ContextImpl.java:360)
D/StrictMode(6996): at android.content.ContextWrapper.getSharedPreferences(ContextWrapper.java:167)
D/StrictMode(6996): at com.google.firebasesandbox.MainActivity.onCreate(MainActivity.java:25)
D/StrictMode(6996): at android.app.Activity.performCreate(Activity.java:6982)
D/StrictMode(6996): at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1213)
D/StrictMode(6996): at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2770)
D/StrictMode(6996): at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2892)
D/StrictMode(6996): at android.app.ActivityThread.-wrap11(Unknown Source:0)
D/StrictMode(6996): at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1593)
D/StrictMode(6996): at android.os.Handler.dispatchMessage(Handler.java:105)
D/StrictMode(6996): at android.os.Looper.loop(Looper.java:164)
D/StrictMode(6996): at android.app.ActivityThread.main(ActivityThread.java:6541)
D/StrictMode(6996): at java.lang.reflect.Method.invoke(Native Method)
D/StrictMode(6996): at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:240)
D/StrictMode(6996): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:767)

So we've discovered here that an activity's onCreate() called getSharedPreferences(), and this method ends up reading the device filesystem. Blocking the main thread like this could be the source of some jank in the app, so it's worth figuring out a good way to move that I/O to another thread.

Now we have a way to check for StrictMode violations during a test, but there's still a few more details to work out.

When should I enable StrictMode?

If you do this immediately when the app launches (in an Application subclass or a ContentProvider), there is a subtle bug in some versions of Android that may need to be worked around in order to avoid losing your StrictMode policies when the first Activity appears.

When should I disable StrictMode?

There might be some times when you know there is a violation, but you can't fix the code right away for whatever reason (for example, it's in a library you don't control). In that case, you might want to temporarily suspend the crashing behavior and simply log it instead. To temporarily suspend the crash, you can define a couple other non-fatal policies and swap them in where you know there's an issue to ignore:

    
private static final StrictMode.ThreadPolicy NONFATAL_THREAD_POLICY =
new StrictMode.ThreadPolicy.Builder()
.detectAll()
.penaltyLog()
.build();

private static final StrictMode.VmPolicy NONFATAL_VM_POLICY =
new StrictMode.VmPolicy.Builder()
.detectAll()
.penaltyLog()
.build();

public static void disableStrictModeFatality() {
StrictMode.setThreadPolicy(NONFATAL_THREAD_POLICY);
StrictMode.setVmPolicy(NONFATAL_VM_POLICY);
}

Similarly, you may not want to track all the possible violations all the time. In that case, you might want to instruct the policy builder to only detect a subset of the potential issues.

Get out there and test your app!

While StrictMode violations are not always the worst thing for your users, I've always found it educational to enable StrictMode in order to find out where my app might be causing problems. In combination with Firebase Test lab, it's a great tool for maximizing the quality of your app.

Share:

Wednesday, 5 July 2017

Test Your Game with Firebase Test Lab for Android

Doug Stevenson
Doug Stevenson
Developer Advocate

Testing your application is a great way to help maximize its quality, and many of you know that Firebase Test Lab for Android has some useful tools for testing Android apps. If you're the type of engineer who likes to maximize test coverage by writing instrumented tests (and regular unit tests), you can send those to Test Lab for execution. Even if you don't like writing tests, you have some options. You can record instrumented tests by interacting with your app using Espresso Test Recorder in Android Studio. And there's almost no effort required at all to run a Robo test that automatically crawls your app.

These tests are helpful for data-driven apps because there are robust test frameworks that understand how to navigate the Android platform widgets used to receive input and display data on screen. However, most games don't work with platform widgets. Games typically take over the screen using their own UI elements, and provide their own touch controls. As a result, it's extremely difficult to write instrumented tests for testing games, and Robo test won't know how to navigate the game at all.

To help deal with these challenges with testing games, the Test Lab team has come up with a way for game developers to test their games effectively across many of the devices that Test Lab offers. It's a new type of test called a Game Loop Test, and it's available today in beta.

How does a Game Loop test work?

If you've seen arcade video games operate, you know that they're always showing something on screen, typically some automated demo of the game, called "attract mode". With Firebase Test Lab, game developers can now use this concept of attract mode to construct test scenarios, and Test Lab will arrange for those scenarios to be invoked in sequence. This gives developers the opportunity to test a wide variety of game levels and situations in a single test run on all the devices provided by Test Lab. If there are any problems with a scenario, you'll find out which ones are problematic, so you can focus your efforts on improving that specific case.

Even better, Test Lab now provides performance data with every report. Of particular interest to game developers is a graph of the rendered frame rate over time throughout the test, tied to a video of the device display. This helps you quickly identify the parts of your game that aren't rendering at an acceptable FPS.

In addition to FPS, there are other performance metrics available for all apps. You'll be able to see your app's CPU utilization, memory usage, and network ingress and egress. Notice how you can jump directly to the point in the video where you're observing performance problems.

If you're a game developer, now is a great time to check out Firebase Test Lab for Android to see what it can do to help the quality of your game.

Share: