Software Engineer REST users rejoice! Conditional requests are here!
How would you increment an integer in the Realtime Database? It's not always as easy as just grabbing a value, adding one, and writing it back again. When multiple users are simultaneously reading, updating and writing the same value, some users' writes could get "lost" due to race conditions.
Let's say you're using the Realtime Database to store upvotes on videos, and two of your users want to upvote the "Silly Cat Video" which currently stands at ten upvotes. Each user reads the value '10' from the database. They then independently increment the value locally and write '11' back to the database. "Silly Cat Video" now only has eleven upvotes when it should have twelve.
How do we prevent this? On a mobile or web client, you could use transactions, which have long been supported in our realtime SDKs. But what if you're using the REST API?
Good news for you: we've just added conditional requests to the REST API! While conditional requests work differently than the realtime transactions, we can still use them to safely perform atomic operations.
Here's how you upvote that cat video the right way: Step 1: Request Data and ETag
Request:
# -i returns headers in the response, -H sets a header curl -i \ -H 'X-Firebase-ETag: true' \ 'https://<your-database>.firebaseio.com/videos/silly_cat_video/upvotes.json'
Response:
HTTP/1.1 200 OK ... ETag: ViJFJowpbyRvgGNPzPJdGeN+mCY= ...
10 // Value at the specified location
Step 2: Write the New Value
Request:
# -X sets the method (defaults to GET), -d provides data to the method curl -i \ -X PUT \ -d '11' \ -H 'if-match: ViJFJowpbyRvgGNPzPJdGeN+mCY=' \ 'https://<your-database>.firebaseio.com/videos/silly_cat_video/upvotes.json'
If no other users have incremented upvotes.json the PUT request will succeed. If another user has, the request will fail and the response will contain the new ETag and data. Use these to calculate a new upvotes value and retry the PUT request.
But wait, what does all this ETag and if-match stuff mean, and how do they work?
The Realtime Database's conditional REST requests use ETagsand the if-matchheader from the HTTP conditional requests standard. An ETag is a short unique identifier for data at a location: if the data changes, so does its ETag. if-match, when set to an ETagvalue, tells the database to only complete a request's operation if the data the request would overwrite matches the provided ETag. Together, these two features enable a Compare and Swap (CAS) paradigm. Compare and Swap uses the two step process we saw in our video upvote example: Step 1: Fetch the location you wish to update along with its ETag. Step 2: Issue a PUT request with new data in the body and the ETag of the old data in the if-match header. If the data at the location changes between Step 1 and Step 2, the database will not complete the operation and will return an error containing the new ETag and data. You can use these values to construct a new PUT request and retry from Step 2.
We have some exciting news for Realtime Database developers on the Flame and Blaze plans: we're increasing your concurrent connections limit from 10,000 to 100,000!
Concurrent connections are the number of users simultaneously connected to your database. For context, it's useful to keep in mind that this number is generally one to three full orders of magnitude lower than your daily active users. To reach 100,000 concurrent users, you usually first need to accumulate tens of millions of daily actives.
Historically, we limited the concurrent connections to 10,000 per database. We requested that you reach out to us to raise this limit, which allowed us to do a quick verification of your app's health. We looked for things like unindexed queries or very heavy reads to make sure your app was ready to scale.
Recent improvements to our infrastructure have made it possible to increase that threshold to 100,000 concurrent connections (but you should still index your queries and never add listeners to the root path!). If you need to support even more concurrent connections, please contact us.
We're excited to bring this change to the Firebase Realtime Database. We've been working hard on improving scalability, and we hope this increase will make scaling more seamless for you. As usual, let us know if you have any questions or feedback. We're always excited to hear from you!
If your app is using Firebase Realtime Database, you've probably gotten a lot of mileage out of its ability to notify your app quickly as changes are made to the database. Your listeners are triggering and receiving new data, users are delighted, and all is well with the world.
However, sometimes listeners may not behave in exactly the way you'd expect when combined with security and validation rules.
There are some important nuances to the way Firebase Realtime Database works, and how those nuances affect the way your listeners are triggered. Let's go over some of those situations, so you can expect the unexpected!
Security Rules with Value Event Listeners
Are you using security and validation rules to protect access to your data? If not, please take a good hard look at that! But if you are using these rules, you can run into some behavior that may seem confusing at first, but is actually predictable, once you understand how the Firebase Realtime Database client libraries work.
All the code samples here will be in Java, because Android is my main thing. But the principles apply to each of the supported platforms, including iOS, web, and JavaScript on the server side.
Imagine you have the following database rules set up:
{ "rules": { ".read": true, ".write": false } }
So, basically, everything is readable and nothing is writable. Your security rules are likely going to be much more specialized, but the point is that some writes will not be allowed at certain locations or under certain circumstances. I'm keeping it simple here, in case you want to experiment with the code samples here in a new project.
Now imagine you have the following tiny bit of data in your database:
ROOT - data - value: 99
You'd expect that a ValueEventListeneron the /data node would give you a snapshot containing a map of the key "value" to the number 99. So, if you executed this code, you'd get a single log statement showing these details:
@Override public void onCancelled(DatabaseError databaseError) { // If we're not expecting an error, report it to your Firebase console FirebaseCrash.report(databaseError.toException()); } }
Pretty straightforward. But imagine you then attempt to change the value from 99 to 100:
HashMap map = new HashMap<>(); map.put("value", 100); dataRef.setValue(map);
Since our security rules prohibit this, we expect to fail. And it does. But one other thing happens that may not be expected. If MyValueEventListener is still registered at the time setValue() is called, it will also be triggered with the new value of 100. Not only that, but the listener will be triggered again with the original value of 99. Your app log might look something like this:
So we see here that the listener got the original value of 99, then the updated value of 100, then an error, then back to the original 99.
Now, you might be thinking, "The security rules should have prevented that change to 100! What gives!" This is a completely understandable perspective. However, it's time to update your expectations with some knowledge about what's really going on here!
The client SDK has no knowledge of the security rules for your project. They live and are enforced on the Firebase server side. However, when the SDK handles the call to setValue(), it goes ahead and assumes that the update will actually work on the server. This is the usual case for code that's been written for a database with a particular set of rules —
the intent is typically never to violate any rules. With this assumption in play, the SDK goes ahead and acts early, as if the write to the database location has actually succeeded. The result of this is the triggering of all listeners currently added to the changed location within the same app process.
OK, so, you might be wondering: if a write can fail, why does the client SDK act early like this? The reasoning is that these immediate callbacks can help your app feel snappy in the face of a poor network connection, and also allows your app to be usable when completely offline. For example, if a user wants to make a change to their profile, why not let them see that change immediately, rather than having to wait for a full round trip to the server? After all, if your code intends to honor the security rules, there should be no problem, right?
In the case where your code does violate a security rule like this, the server notifies the app that the update actually failed at that location. The logical thing to do, at this point, is trigger all listeners at that location with the original data, so the UI of your app can regain consistency with known values from the server.
Given all this context on how security rules works, let's look at another scenario.
Security Rules with Child Event Listeners
Child event listeners are different from the value event listeners described above. A ValueEventListeneras shown above gives you the entire contents of a particular location, every time any part of it changes, whereas a ChildEventListenergives you callbacks for individual child nodes under a location whenever one of those children is added, changed, moved, or removed.
For this example, let's use the same security rules as before, with everything readable and nothing writable:
{ "rules": { ".read": true, ".write": false } }
Now, let's say you have a node in your database called /messages, where you want users to be able to push new message content to be shared with others:
private class MyChildEventListener implements ChildEventListener { @Override public void onChildAdded(DataSnapshot dataSnapshot, String s) { Log.i("**********", "childAdded " + dataSnapshot.toString()); }
In this code, we have a ChildEventListener added on /messages, then we're trying to add a new child object into a location determined by some generated push id. Of course, we expect this to fail because of the security rules. But, let's look at the log to see what actually happens if we execute this code:
I/**********: childAdded DataSnapshot { key = -KTfacNOAJt2fCUVtwtj, value = {key=value} } W/RepoOperation: setValue at /messages/-KTfacNOAJt2fCUVtwtj failed: DatabaseError: Permission denied I/**********: childRemoved DataSnapshot { key = -KTfacNOAJt2fCUVtwtj, value = {key=value} }
We see that the client library immediately triggers the onChildAdded method with the new child object under /messages, then logs an error, then triggers the onChildRemoved callback with the same object.
If you read through and understood the prior example, this one should be a little less surprising. The Firebase client SDK is again acting earlyin response to the call to setValue() and assuming that the write will success. Then, after the write fails because of the security rules, it attempts to "undo" the add that failed. This ensures that the app's UI can remain up-to-date with the correct child values, assuming that it has implemented onChildRemoved correctly.
The behavior of the Firebase client library in the face of violated security rules should be more clear now, but you might still be wondering how you can detect if a violation occurred. It may not be adequate for your app to simply reverse the effect of the write. In fact, you may even want to know if and when that actually happens, as it could be considered a programming error. This brings me to the next point.
Detecting Write Errors
In the examples above, it can be very difficult to tell if your call to setValue() failed at the server just by looking at the listener callbacks. If you want to detect failure, you'll need a bit of extra code to respond to that event. There are two ways to do this. First, there is CompletionListenerthat you can pass to an overload of setValuethat gets notified of errors. Alternatively, you can also use the Play Services Task API by using the Taskobject returned by setValue. I'll prefer a Task here, because it has built-in protections against Activity leaks (note the first argument to addOnCompleteListeneris an Activity instance):
Task task = messageRef.setValue(map); task.addOnCompleteListener(MainActivity.this, new OnCompleteListener() { @Override public void onComplete(@NonNull Task task) { Log.i("**********", "setValue complete"); if (!task.isSuccessful()) { Log.i("**********", "BUT IT FAILED", task.getException()); FirebaseCrash.log("Error writing to " + ref.toString()); FirebaseCrash.report(task.getException()); } } });
When the write of the value completes, with either success or error, the OnCompleteListenerregistered to the Task will be called. If it failed, I can check the Task to see if it was successful and deal with it as needed. In the above code, I'm choosing to report the error to Firebase Crash Reporting, which can help me determine if and where I made a mistake in my code or security rules. It's probably a good idea to always report your write failures like this, unless you fully expect that a write could legitimately fail, under normal circumstances, to a security rule.
When there is an update to a location that also has active listeners in the same process, the flow of data through the process goes like this:
Immediately call all relevant listeners with the new value
Send the update to the Firebase server side
Check security rules for validity
If a security rule was violated, notify the client SDK
Roll back the change in the app by calling relevant listeners again to back to the original state
Using this knowledge, it's possible you may have reset your expectations to expect the unexpected for your listeners! Were your expectations changed? Let me know in the comments below! And, if you have any programming questions about Firebase Realtime Database, you can ask us on Stack Overflow with the firebase-databasetag. For more general questions, you can ask on Quora or use the firebase-talkGoogle Group.
If you like, follow me on Twitter as CodingDoug, and don't forget to check out our YouTube channel for Firebase tutorials and other shows.
With Firebase, we've been working towards a world where developers don't have to deal with managing servers and can instead build web and mobile apps with only client-side code. However, there are times when you really do need to spin up your own server. For example, you may want to integrate with a third-party API (such as an email or SMS service), complete a computationally expensive task, or have a need for a trusted actor. We want to make your experience on this part of your stack as simple as it is on the front-end. Towards that aim, we announced the Firebase Admin SDKs for Node.js and Java at the Firebase Dev Summit in Berlin earlier this week.
What are the Admin SDKs?
The Firebase Admin SDKs provide developers with programmatic, second-party access to Firebase services from server environments. Second-party here refers to the fact that the SDKs are granted elevated permissions that allow them to do more than a normal, untrusted client device can. The Admin SDKs get these elevated permissions since they are authenticated with a service account, a special Google account that can be used by applications to access Google services programmatically. The Admin SDKs are meant to complement the existing Firebase web and mobile clients which provide third-party, end-user access to Firebase services on client devices.
Some of this may sound familiar for those of you who have used the existing Firebase Node.js and Java SDKs. The difference is that we have now split the second-party (aka "admin") and third-party (aka "end-user") access use cases into separate SDKs instead of conflating them together. This should make it easier for beginners and experts alike to know which SDK to use and which documentation to follow. It also allows us to tailor the Admin SDKs towards server-specific use cases. A great example of this is the new user management auth API which we will go into in the next section.
What can the Admin SDKs do?
The Admin SDKs for Node.js and Java offer the following admin capabilities that already existed in the prior server SDKs:
In addition, the Node.js SDK brings some exciting new functionality:
Admin API for managing Firebase Auth users, including the ability to fetch user data, create new users, update user properties, and delete users. You can do all of this without a user's existing password and without worrying about rate limiting. This has been a heavily requested feature from developers for years now and we are excited to finally put it into your hands.
The best place to start is with our Admin SDKs setup guide. The guide will walk you through how to download the SDK, generate a service account key file, and use that key file to initialize the Admin SDK. Thanks to our new Service Accounts panel in your Firebase Console settings, generating service account keys should be a breeze.
What's next for the Admin SDKs?
This is really just the beginning for the Admin SDKs. We plan to expand the Admin SDKs across two dimensions. Firstly, we want to provide Admin SDKs in more programming languages, allowing you to write code in the language you feel most comfortable. Secondly, we plan to integrate with more Firebase services, including adding support for services like Firebase Cloud Messaging and bringing the new user management API to Java.
Would you like us to build an Admin SDK in a particular language? Do you want the Admin SDKs to support a certain Firebase service or feature? Let us know in the comments below or by sending us a note through our feature request support channel.
We are excited to expand our first-class support for backend developers in the Firebase ecosystem. Stay tuned for more to come in the future!
Private Backups is a good solution for backing up, scripting, and gathering analytics by regularly exporting your entire Database. Previously, setup involved a bit of back and forth between the user and Firebase Support. We are now offering a self-service user interface for Blaze customers in the Firebase console that seamlessly enables daily backups of your Firebase Database to a Google Cloud Storage bucket.
To get started, visit the new “Backups” tab in the Database section of the Firebase console, and the wizard will guide you through setting up your automated backups.
To save on storage costs, we enable Gzip compression by default, and you can choose to enable a 30-day lifecycle policy to have backups older than 30 days automatically purged.
When the setup process is complete, you will have a Google Cloud Storage bucket. On a daily basis, we will backup your Database’s application data and rules in JSON format to your bucket. You can view the status and history of all your backups directly in the Firebase console. There is also a “Manual Backup” button which lets you instantly enqueue a current backup of your database and rules, and is incredibly useful to take specific timed snapshots or as a safety action before you perform any code changes.
Here are three good reasons to start using automated backup today:
Offline Scripting
Automatic Database Backups let you crawl, analyze, and perform large scripted actions against your data without affecting the realtime performance of your customers connecting to the Database.
Historical Data Analysis
With historical snapshots of your database, you can retroactively analyze your data, look for trends, and make product decisions based on past data points.
Recovery From Corrupt or Lost Data
With Automatic Database Backups, you can easily restore your data in the event of data corruption or loss. To restore your database, simply download1 a backup file to your local machine from the backups tab. You can then click the "Import JSON" button2 under the data tab to restore your Database to a previously backed-up state.
We are continuing to streamline the experience for Firebase developers, and this new release gives developers more control over their data while reducing the need to go through support processes.
Notes
With Gzip enabled, you will need to decompress your database data before importing back into Firebase. ↩
For very large Databases, it is more efficient to contact support for data restoration from our own regular backups, rather than through the JSON files of Automatic Database Backups. ↩
For most developers, building an authentication system for your app can feel a lot like paying taxes. They are both relatively hard to understand tasks that you have no choice but doing, and could have big consequences if you get them wrong. No one ever started a company to pay taxes and no one ever built an app just so they could create a great login system. They just seem to be inescapable costs.
But now, you can at least free yourself from the auth tax. With Firebase Authentication, you can outsource your entire authentication system to Firebase so that you can concentrate on building great features for your app. Firebase Authentication makes it easier to get your users signed-in without having to understand the complexities behind implementing your own authentication system. It offers a straightforward getting started experience, optional UX components designed to minimize user friction, and is built on open standards and backed by Google infrastructure.
Implementing Firebase Authentication is relatively fast and easy. From the Firebase console, just choose from the popular login methods that you want to offer (like Facebook, Google, Twitter and email/password) and then add the Firebase SDK to your app. Your app will then be able to connect securely with the real time database, Firebase storage or to your own custom back end. If you have an auth system already, you can use Firebase Authentication as a bridge to other Firebase features.
Firebase Authentication also includes an open source UI library that streamlines building the many auth flows required to give your users a good experience. Password resets, account linking, and login hints that reduce the cognitive load around multiple login choices - they are all pre-built with Firebase Authentication UI. These flows are based on years of UX research optimizing the sign-in and sign-up journeys on Google, Youtube and Android. It includes Smart Lock for Passwords on Android, which has led to significant improvements in sign-in conversion for many apps. And because Firebase UI is open source, the interface is fully customizable so it feels like a completely natural part of your app. If you prefer, you are also free to create your own UI from scratch using our client APIs.
And Firebase Authentication is built around openness and security. It leverages OAuth 2.0 and OpenID Connect, industry standards designed for security, interoperability, and portability. Members of the Firebase Authentication team helped design these protocols and used their expertise to weave in latest security practices like ID tokens, revocable sessions, and native app anti-spoofing measures to make your app easier to use and avoid many common security problems. And code is independently reviewed by the Google Security team and the service is protected in Google’s infrastructure.
Fabulous use Firebase Auth to quickly implement sign-in
Fabulous uses Firebase Authentication to power their login system. Fabulous is a research-based app incubated in Duke University’s Center for Advanced Hindsight. Its goal is to help users to embark on a journey to reset poor habits, replacing them with healthy rituals, with the ultimate goal of improving health and well-being.
The developers of Fabulous wanted to implement an onboarding flow that was easy to use, required minimal updates, and reduced friction with the end user. They wanted an anonymous option so that users could experiment with it before signing up. They also wanted to support multiple login types, and have an option where the user sign-in flow was consistent with the look and feel of the app.
“I was able to implement auth in a single afternoon. I remember that I spent weeks before creating my own solution that I had to update each time the providers changed their API” - Amine Laadhari, Fabulous CTO.
Malang Studio cut time-to market by months using Firebase Auth
Chu-Day is an application (available on Android and iOS) that helps couples to never forget the dates that matter most to them. It was created by the Korean firm Malang Studio, that develops character-centric, gamified lifestyle applications.
Generally, countdown and anniversary apps do not require users to sign-in, but Malang Studio wanted to make Chu-day special, and differentiate it from others by offering the ability to connect couples so they could jointly countdown to a special anniversary date. This required a sign-in feature, and in order to prevent users from dropping out, Chu-day needed to make the sign-in process seamless.
Malang Studio was able to integrate an onboarding flow in for their apps, using Facebook and Google Sign-in, in one day, without having to worry about server deployment or databases. In addition, Malang Studio has also been taking advantage of the Firebase User Management Console, which helped them develop and test their sign-in implementation as well as manage their users:
“Firebase Authentication required minimum configuration so implementing social account signup was easy and fast. User management feature provided in the console was excellent and we could easily implement our user auth system.” - Marc Yeongho Kim, CEO / Founder from Malang Studio
This year's Google I/O was an exciting time for Firebase. In addition to sharing the many innovations in our platform, we also hatched a time-traveling digital fish named Firebass.
Firebass is an Alternate Reality Game (ARG) that lives across a variety of static web pages. If you haven’t played it yet, you might want to stop reading now and go fishing. After you’ve caught the Firebass and passed the challenge, come back -- we’re going to talk about how we built Firebass.
How we began
We partnered with Instrument, a Portland-based digital creative agency, to help us to create an ARG. We chose ARG because this allowed us to utilize developers’ own software tools and ingenuity for game functionality.
Our primary objective behind Firebass was to make you laugh, while teaching you a little bit about the new version of Firebase. The payoff for us? We had a blast building it. The payoff for you? A chance to win a free ticket to I/O 2017.
To begin, we needed to establish a central character and theme. Through brainstorming and a bit of serendipity, Firebass was born. Firebass is the main character who has an instinctive desire to time-travel back through prior eras of the web. Through developing the story, we had the chance to revisit the old designs and technologies from the past that we all find memorable -- as you can imagine, this was really fun.
Getting started
We put together a functional prototype of the first puzzle to test with our own developers here at Google. This helped us gauge both the enjoyment level of the puzzle and their difficulty. Puzzle clues were created by thinking of various ways to obfuscate information that developers would be able to recognize and manipulate. Ideas included encoding information in binary, base64, hex, inside images, and other assets such as audio files.
The core goal with each of the puzzles was to make them both logical but not too difficult -- we wanted to make sure players stayed engaged. A bulk of the game’s content was stored in Firebase, which allowed us to prevent players from accessing certain game details too early via inspecting the source code. As an added bonus, this also allowed us to demonstrate a use-case for Firebase remote data storage.
Driving the game forward
One of our first challenges was to find a way to communicate a story through static web pages. Our solution was to create a fake command line interface that acted as an outlet for Firebass to interact with players.
In order to ground our time travel story further, we kept the location of Firebass consistent at https://probassfinders.foo/ but changed the design with each puzzle era.
Continuing the journey
After establishing the Pro Bass Finders site and fake terminal as the centerpieces of the game, we focused on flushing out the rest of the puzzle mechanics. Each puzzle began with the era-specific design of the Pro Bass Finders home page. We then concepted new puzzle pieces and designed additional pages to support them. An example of this was creating a fake email archive to hide additional clues.
Another clue was the QR code pieces in puzzle 2.
The QR codes demonstrate Firebase time-based read permissions and provide a way to keep players revisiting the site prior to reaching the end of puzzle 2. There were a total of three pieces of a QR code that each displayed at different times during the day. It was really fun and impressive to see all of the different ways players were able to come up with the correct answer. The full image translates to ‘Locating’, making the answer the letter ‘L’, but many players managed to solve this without needing to read the QR code. You're all smart cookies.
Final part of the Puzzle
Puzzle 3 encompassed our deep nostalgia for the early web, and we did our best to authentically represent the anti-design look and feel of the 90s.
In one of the clues, we demonstrated Firebase Storage by storing an audio file remotely. Solving this required players to reference Firebase documentation to finish writing the code to retrieve the file.
<!-- connect to Firebase Storage below --> <script> console.log('TODO: Complete connection to Firebase Storage'); var storageRef = firebase.app().storage().ref(); var file = storageRef.child('spectrogram.wav');
// TODO: Get download URL for file (https://developers.google.com/firebase/docs/storage/web/download-files) </script>
The finale
While the contest was still active, players who completed the game were given a URL to submit their information for a chance to win a ticket to Google I/O 2017. After the contest was closed, we simply changed the final success message to provide a URL directly to the Firebass Gift Shop, a treasure in and of itself. :)
Until next time
This was an unforgettable experience with a fervently positive reaction. When puzzle 3 unlocked, server traffic increased 30x! The community response in sharing photos, Slack channels, music, jokes, posts, etc. was incredible. And all because of one fish. We can’t wait to see all the swimmer winners next year at I/O 2017. Until then, try playing the game yourself at firebase.foo. Thank you, Firebass. Long may you swim.