Video: Google I/O 2013 – Intense Gaming
CHRIS ELLIOTT: –on to the next talk on the Cloud Platform track. This is actually an amazing turnout considering the Glass Dev section is right across the street. When I found out I was scheduled to go right against the Glass sessions, I was planning to turn into a small fireside chat with a few people, but this turnout blew me away. But we have quite a good session in here planned up in line for you, so please don't leave. I'm glad nobody got up and ran to the back. Oh, the Glass session's happening now? So without any further ado, let's get this started.
I remember being a game developer many years ago, and I was thinking about this recently. And it was really funny because the hardest thing for me to find was dedicated game servers to run my game engine and test with players. I would be trolling IRC at late hours of the night, just trying to find somebody willing to host my game and get 32 friends on it to really stress test it. It was extremely hard to find. And it's interesting today that that is no longer a problem. You have a vast amount of computing resources available for building out amazing games, and now the challenge is really, how do you create those games and create them in a way that's really going to scale as the number of users playing your game scales? And at the end of the day, it really comes down to building in a way that when your game becomes an unexpected success– there's just massive growth, 20 fold over what you expected– that you can survive that onslaught scale and make sure everyone can get in and have a really great experience on your platform.
And that's really what we're talking about with this session is how to make that possible on the Google Cloud Platform. So the game plan starts off with a very high level Compute Engine and App Engine and Google Cloud Platform overview, just highlighting the fundamental building blocks that we'll be working with throughout the session. Up next is looking at mobile gaming architecture that really shows you how all these pieces fit together to solve a lot of common problems mobile developers face when they're building out games. And then up next, what better than to hear it from people who have done this themselves and scaled to a huge user base? And so some great guys from FreshPlanet are going to come up and talk about how they used this platform to power their game "SongPop." And then finally, wrapping it up with a dedicated game server that really targets the real time gaming where you need tens of thousands of virtual machines running all of your game servers. So that's really what we have in store over the next 45 minutes.
So many of you have probably seen this slide if you attended the Cloud Platform keynote, and it really highlights all of the amazing technology we've exposed. And really, it's amazing how you connect it. But as a developer, the biggest challenge is, what do I need to know here? What is relevant? How do all the pieces connect together? And furthermore, how do I augment all these other Google services, such as the Play Games that was announced yesterday? What value can that bring to my app? So what's really interesting and what continues to blow me away is just the scale of the Google Cloud Platform. We announced that we have over three million active applications yesterday and over 300,000 developers every month. And just every day, we seem to run into really amazing game studios pushing the limits of the technology in creating these next generation gaming experiences.
So this is just a sample of the game studios that are building on the platform. And what's really interesting, it's a huge spectrum. You're looking at AAA Studios all the way down to small indie studios, and they're all leveraging the platform to deliver these next generation experiences. And what's even cooler is it's not just one game. They really get hooked on App Engine, and it becomes a way of building out their games. They understand the platform, they know how to scale on Google, and they trust Google to scale based on past experiences they've had. For example, EA Games has launched over five games on the platform from "Tetris" to "The Simpsons Game" to "Yahtzee." You don't even realize what's powering these games because it just scales seamlessly as these games take off and you see all of your friends playing them. And another one that really blew me away was Epic Games. I had no idea "Infinity Blade 2," the compelling social features were all powered by App Engine– no idea until I was talking to them at GDC and they mentioned this, and it's just absolutely amazing.
And what blew away the most about this example was the fact that it was done by mostly one guy in six weeks. Could you imagine having to build out the back end for "Infinity Blade 2"? You have six weeks, you're one guy, and you know that tens of millions of people are going to download this game. That's pressure, and that's really when they turned to App Engine because they know and trust it to scale and deliver and allow them to really launch and have a great success. They also used it for "Vote!! The Game" as well. That was a very timely piece, but it was also pretty cool. Check it out as well for "Infinity Blade" meets election politics. And looking at the core fundamental aspects of building on the Google Cloud Platform, first one is App Engine. Most people know what this is. It's been around for five or six years, and really that is the Platform as a Service that's easy to build, easy to deliver, easy to deploy, and easy to maintain your application. You write it in a language of your choice that's provided, and then you can use this platform to really scale up with respect to your user base.
Another real, foundation building block of gaming solutions is Compute Engine. It's looking at that scenario where you need your own open and flexible platform to run any code, any packages, any software, to run your custom game engine, to run some large game engines. If you want to run NoJS, it provides that flexibility. And I'm going to touch a little bit on really what we've seen Compute Engine being great for for developers and what kind of experiences they've been having on the platform with Compute Engine specifically. And time and time again, it is the consistent performance and the amazing network that has blown developers away and pushed their imagination of what's actually possible on Cloud Platforms. Here's just one of them.
Actually, Ed's in the audience right now. You can talk to him more about this, that I didn't just make this quote up. So Chrome JAM, you may have heard about this. It was a project that was basically you log in, jam out with four friends, real time audio streaming, and it had extremely tight latency requirements. And it wasn't even possible on some of the other cloud platforms, but it was possible on Compute Engine. So when I got this quote that your latency's crazy low, pat the team on the back, definitely fired it off to the networking team. And they love seeing feedback like this because it really justifies all the hard work we put into the technology over the many, many years. Also, you may have seen this MapR Terasort World Record. The MapR broke the Terasort World Record on Compute Engine. Amazing. What's really cool about this for game developers is the fact that they broke it in a cloud-based environment, and the past record was held by dedicated hardware in their own data center.
So this really challenges the expectations as a developer if you're building out a game. OK, I need 1,000 servers in the East Coast to support my traffic there, 1,000 in Europe. You can really now leverage the Cloud Platform and start to scale up where your user base is. And the key message here is if you tried cloud platforms in the past for running these game engines, and they just haven't been able to scale or have the latency or performance you expect, check our Compute Engine. You'll be happily surprised. And this is just one of the many examples that show how that performance is delivering real results. And finally, it really can be summed up in a nice, neat little package with a little bow. Kunal from Wizard Ops, they moved their application over to Compute Engine and immediately, they went, "Everything is just noticeably faster in the application." And we're not talking about going in the logs and seeing 10 milliseconds.
It's actually in the application, playing the game, firing off requests, doing all the actions, everything just responded quicker, and the performance blew them away from what they were expecting based on past experiences. So that is Cloud Platform in a nutshell, and now we're going to jump into an architecture that highlights how to piece together all of those components. And with that, we always try to highlight the key design requirements a lot of developers face when building out these applications. Firstly, iOS, Android devices– and if you're building a game now, quite often you're looking at multi-platform. And the other really big important thing is being scalable to millions of users, but not just being able to hit a million users, ten million, 100 million users, but scaling in a way that makes sense so you're not over and under provisioned, and you're not spending a huge amount on servers when you only have five players. And at the end of the day, it's really about creating an engaging social game that just grows virally and gives you some really great growth. And actually, FreshPlanet will talk a bit more about things they did with that. And whenever we do one of these reference architectures, we love to show an application that highlights it is really straightforward to get up to speed and start running and piecing together all these components.
And so with that, we created a sample application called Griddler which we're going to open source very soon, and it just highlights how you can build a mobile game very quickly on the platform. And so if I switch over to the overhead. It works. So essentially, you join in this game. It's an asynchronous-based mobile real game. And so you can go there, you can challenge your friends, you can play by yourself. And so if I jump in and I go to Challenge An Opponent, you'll see that I only have one friend, but Greg, he's my best friend. We play this game all of the time. And so you go in here, and you challenge him to a game. So this will basically then hit App Engine, send off a push notification to his device. And then once he accepts that device, it'll start a count down, and we'll be competing in real time against this game. It's kind of unfair because Greg worked very closely developing the game and knows all of the answers to the riddle, where I've only made it through the first three. And so once he accepts the push notification that makes it to his device, that should start, assuming we don't have any unexpected Wi-Fi challenges and so forth.
Anything on your device yet? Well, what you will see– we'll try one more time. I don't want to have one of these mobile demos that goes down in flames. We seem to have really bad experience with doing mobile demos in Cloud Platform. Always, the unexpected stuff happens regardless of how much you'll test. But you'll notice that I'm actually plugged into the power on the phone, so we learn from our mistakes very quickly at Google for anybody who was at the keynote. So it looks like the phones in this room and the wireless isn't quite working right, so let's jump into Single Player. And so this is exactly what you see. So you're hitting App Engine, you're getting requests for all of the game boards, and you jump in.
And so you get something like, what has to be broken before it can be used? I know the answer because I've done this one a few times. Egg. Feed me and I live, yet give me a drink and I die. Fire. Let's see. Where's fire? Right here. And so you go through, and you're competing against your friend to get the quickest time, the highest score, et cetera. What has four legs but only one foot? I didn't know the answer to this, and I was thinking about this for 10, 15 minutes. One of the guys in the back setting up for all the AV, they're rock stars. They figured it out. We're just setting up here and out of nowhere, the guy yells, it's bed, bed! What are you talking about? It makes perfect sense, bed. So that's kind of what we're looking at here. But this next one honestly, I don't know what the answer is.
I really don't answer is, so we're kind of blocked here unless anybody has any idea. I fly straight and on track, I take off when pulled back. Yeah? AUDIENCE: [INAUDIBLE]. CHRIS ELLIOTT: Arrow. Is it in the game board? Oh yeah. All right, you're a rock star. What runs and never gets tired? AUDIENCE: [INAUDIBLE]. CHRIS ELLIOTT: Water? Wow, you guys are rock stars. There's water. What kind of tree can you carry in your hand? What's that? AUDIENCE: [INAUDIBLE]. CHRIS ELLIOTT: A palm tree? I tried palm. It didn't work before. Well anyways, it goes on like this. We could spend the entire session doing this. Show of hands, who wants to do riddles the entire session, or who wants to see some architectures? OK, there was a surprising number of hands for riddles, but it might not be great for anybody viewing at home after the fact.
So that's the kind of game we're looking at, so let's jump back over to the Mac. And when building out a game like that, the core building block really is the mobile application, creating your native application, iOS, on Android, whatever devices you want. And really where the magic begins is when you connect it to App Engine. Cloud Endpoint is an amazing piece of technology for creating a RESTful API and seamlessly connecting your mobile applications to your front ends running on App Engine. It provides RESTful APIs and strongly typed libraries for your devices that make it very simple to call that from code natively. I had the misfortune of having to do this before Cloud Endpoints existed, creating a RESTful API with full authentication, and it is not easy to do that, and Cloud Endpoints makes everyone's lives a lot easier now. And so what does that actually look like? It really is dead simple to put together an application that calls Cloud Endpoints.
You add this annotation to declare the path you would like and the variables, and then you'll see essentially within App Engine in Java, you define the function, you define the variables you're looking at, and then what will be generated is this client library– for example, shown here for Android– that just makes it very simple to call gameBackend.sendInvitation, and that's it. One of the really cool things here you may not have noticed is the fact that we have a user parameter in App Engine to use, but there's nothing we're sending actually from the client application. And that's populated automatically if you need that for application and making sure the user is who they say they are, and you have the full authentication baked in. Really, really cool stuff, and that all uses Google's API technology.
The other aspect is looking at pushing notifications out to the device, such as this push notification that I sent to Greg. And this is really where Google Cloud Messaging provides all that functionality, and you can even leverage some of the stuff that was announced yesterday. But another thing that's really cool here is Apple Push Notification works from App Engine. So some people in the room might have tried this a year ago and realized it wasn't possible very quickly. But now, it is possible because the socket's API makes it seamless to integrate Apple Push Notification Service directly with App Engine and call that without much effort on your end. And so it really is seamless. I mean, within App Engine, you import the required libraries and so forth. And then from there, it's the standard exactly what you'd expect, the push.Alert, and you send the device IDs, the messages, et cetera.
So we actually are going to have a paper published very soon to show you how to build that out on App Engine, and also a sample application to give you something to get up to speed very quickly. So the only nuances that come into play here are just working with Apple Push Notification Service. For example, they don't like you opening many, many socket connections, so you'd be looking at using something like maintaining a socket connection in the back end that lives around forever and doesn't terminate after every request. And then you can use mechanisms like the task queues to really batch those up and send them through. So all the components of App Engine really provide the scalable framework to allow you to scale up to sending out tens of millions of push notifications. And so one of the secret sauces of App Engine– one of the many, many secret sauces. I mean, it's a delicious thing full of crazy secret sauces, but the Datastore, in my opinion, is one of the biggest things that really gives App Engine, makes it a great place for developers to work. It's a NoSQL database that just scales to ridiculous size.
And when you combine that with Memcache, you can really scale up to 80 million users without having any hiccups at all in your database layer. And so with that, for Griddles for example, all of those game boards are frequently accessed and requested, it makes sense to cache those in Memcache. I mean, Memcache isn't an infinite service, and you need to use it wisely. But those are being requested for every single player, and it's a perfect use case. However, the Datastore is a great place to store all of those user scores, profiles, et cetera, all of the games, all of the challenges. And of course, you'll keep all of the boards in there long term and just refresh them in case the Memcache gets evicted. So that's one example of how you can think about using Datastore and Memcache when you're building out these mobile applications.
And then the other aspect really is the Tasks Queued, Schedule Tasks. These we kind of highlighted when talking about Apple Push Notification Service. And Schedule Task, for example, would be great if you want to go through daily and send out a notification to anybody who hasn't played the game in a week just saying, hey, we have new content. Would you like to try, or a lot of these things you see mobile applications doing. And up next, a lot of times, you need to have an online object store to store video or pictures or a lot of this rich content that users expect. You may have seen this in the Cloud Platform Overview where we're storing video on Cloud Storage, and this is really straightforward to connect to App Engine, connect to your mobile devices, and start storing all that content here. And if you're looking at things like using images, the Image service is a great way of rescaling any images a user uploads so it matches whatever UI you have set up with your mobile application.
And finally, Compute Engine. I mean, it's amazing to be able to fork out to Compute Engine any of these batch processing workloads or any time you want to run any code at all that doesn't necessarily work within App Engine, and that is really where Compute Engine is a great use case in this example for mobile applications. Although you want to leverage App Engine for a lot of the scalability aspects– that'll get you up to speed the quickest– Compute Engine is there to back you up when you need to run your own Linux virtual machines and custom software packages and so forth. The example we really like to talk about is video transcoding, for example. You have videos in the Cloud Storage, pulling them out, doing the transcoding, and putting them back in Cloud Storage. And then the Task Queue is really the key mechanism to orchestrate that communication between Compute Engine and App Engine. And that is also very seamless to set up. I realized during this presentation, I say "seamless" many, many times, but it really is seamless. Talk to anybody who has built on the platform and created these massive games, and they will echo the exact same sort of sentiments.
So when you create the instance, all you have to do to set up Task Queue is really straightforward. You just specify the service scope to give access to the Task Queue within that project. From there, if you were in the instance on Compute Engine, you'll have access to the metadata with the access token or to call out like that. Otherwise, if you're using things like the Client Library, it's very seamless to go in and just one line of code, to configure that, and then from there, you can start calling Task Queue and can perform a lot of operations there. And those operations are similar to what you'd expect. I mean, you can pull a lease down on a task, do a bunch of processing work, and then delete that task once everything's completed. And one of those completion steps might include putting another task back on the queue where App Engine can be pulling that information back down and doing some more processing, notifying the user, et cetera.
And finally, if you're looking at things like autoscaling, the Task Queue Stats is a great place to go there as well. So if you see you have 1,000 videos waiting to be transcoded, increase the number of instances. That's just one of the many ways you can scale on Compute Engine, and you can really have the flexibility to tweak that to whatever kind of techniques you're familiar with using. So the last aspect of the solution is BigQuery. BigQuery is absolutely amazing. I was talking to a customer a couple days ago, and they thought it was voodoo magic because it's real time analytics for massive data sets. And so they ran a query over five billion rows, and it returned in three seconds. That didn't make any sense to them at all.
This is impossible. Something must be wrong. But actually, it is working correctly. It's based on proven, bullet tested Google technology. And it's really great for mobile games. Almost every studio we talked to that is building on the platform and has scaled up to millions of users uses BigQuery for getting a lot of really great insight into their application. A great example of how you can use it is you have every single action per user per day. It's a great opportunity to run a MapReduce over that job and aggregate it down to a level that you really care about. For example, total actions per user per day instead of every single action. At that point, you can put it into BigQuery and start asking really insightful things, such as, what happens to users who stop playing for a week, and we send them a promotion? Do they come back? Or if I have a double experience weekend, do people spend more? Do people play longer? It's a really powerful component for getting a lot of this insight that is key to building a mobile application.
And finally, you've actually come to the best Glass session in my opinion, although I might be a little bit biased. When looking at building out apps on Glass, I mean, you have the Mirror API, you can do native Android applications. But really, it's the back end where the magic happens. When you're creating the next generation of applications powered by Glass, Google Cloud Platform is the best place to go and build those applications out because you can leverage all of this technology. And when you only have a couple of developers with devices, there's not going to be a lot of cost, but once it takes off, you'll be able to scale up massively. And so really, you can use things like Cloud Endpoints, you can connect it through Mirror API. You may have noticed on the Mirror API, for example, App Engine is a great place to go for Java and Python, and so all of these techniques are directly applicable to Glass. So this is really how you build Glass applications.
All those people on the other side are in the wrong session. Don't tell them I said that. Might not work out too well. So what better way than to actually hear about how you build on a platform and that it's not just voodoo magic I'm talking about, that it actually is possible to create a very successful application? I'd like to welcome to the stage Olivier and Alexis from FreshPlanet who have built this amazing game, "SongPop," and they're going to tell you all about it. Thanks, guys. [APPLAUSE] OLIVIER MICHON: So I am Olivier, and this is Alexis. We are not yet on Google Glass, but hopefully soon. So we are FreshPlanet. We are a small company, we're a start-up in New York City around 20 developers right now, and we are focusing on making mobile social games on app. Something that sets us apart, I think, is that we are very small teams, so we usually have between three and four developers per project. But at the same time, we want to have a very large scale, so lots and lots of users, so we have to be very efficient and find the right technology that allows that, and this is one of the reasons why we use Google Cloud Platform.
About one year ago, we released "SongPop." I don't know if you guys have played it a little. We wrote a few months ago a blog post on the Google Cloud Blog, and Chris was nice enough to invite us to share a bit more detail with you. So for those of you who don't know, "SongPop" is a very simple game. At its heart, it's really you have five song samples that get streamed to you. You have to guess either the song title or the artist. This is on iOS, Android, or on the web. The twist to it is that it's purely muliplayer. You never play alone. You're always competing against someone else. It can be a stranger or one of your friends. I think we did something right with this game because this is what our growth looks like. We did absolutely no marketing at all. And this is a million daily active users. We released around May of last year. By June, we were already at two million daily active users with a curve that looks like the famous hockey stick, only it keeps growing and growing as we reach more users across many different countries. We plateaued around September of last year with five million daily active users, and at this time, we were the first game in the App Stores in many countries including the US, the UK, and Australia.
In terms of scale, what it represents when you're at this point, it's 300 million requests a day on your infrastructure. It's three billion Datastore operations per day. We were streaming around 20 terabytes a day of songs and album covers for the music. So let's dig a bit and see how we structure that and which of the Google Cloud products we used to reach this scale. On the left, you can see that we have those three different clients that we support, so iOS, Android, the web. They're all talking to our App Engine back end written in Python with the same simple JSON API. Then they fetch all the media contents, so the song samples and the album cover on Google Cloud Storage. And we have to push data back to our users when we release new playlists, when we think they might be interested, when they have a new challenge. We also have a chat enabled in the game so when you receive a new message.
For that, we use Google Cloud Messaging for Android. We use Apple Push Notification with Socket. We use a channel API on the web. We integrate deeply with Facebook for friend requests on all of the API. But really at the centerpiece of it all is our App Engine code written in Python, and it integrates with all the different pieces of the Cloud Platform– the Datastore, Memcache, Task Queue, we use all of that. So let's see what kind of logic we have to run because at first, it seems like a very simple game somewhat, but I think we do a few interesting things. On the left, you can see the matching. So you can, of course, challenge an opponent on Facebook with username and email, but we manage to run more interesting queries on App Engine.
For instance, we have a feature called Best Match. What it does, it's going to try to find an opponent for you that wants to play with strangers, that's currently active, that's close to you geographically– so we look at the state and even the city level– and then we match you on your music taste because you will have a way better experience if you play with someone that likes the same kind of music because you're going to go back and forth with music you both like. So it was very important. After we released this feature, we looked at the metrics, and it was very beneficial for us compared to just random matching to take user music tastes into account. Those kind of things can run on App Engine even if they are very complicated. On top, we also have lots of Playlists to manage and lots of quizzes to generate. We have to create hundreds of playlists with up to 500 songs each, and when you generate a playlist, we have to take into account you as a player and your skills in this playlist, but also the person that's going to receive it.
Based on that, we have to balance and pick the right set of songs that we want. But I think the most interesting part is the [INAUDIBLE] button because most games, their results are on one user in terms of data, in terms of scalability. But SongPop is different because it's purely one versus one, so most of our data is between two users. It's what we call the Player versus Player, or PvP. It means that our database is really a graph structure, and so we store the current quiz there, we store a bunch of stats on your histories of scores and playlists sent. We store the Chat messages. So it's a bit of particularity and we are pretty heavy users of the back end, both in terms of data and logic. But to dig even deeper on that, I'm going to let Alexis talk about it. ALEXIS HANICOTTE: So let's first start with how we define our models.
That's pretty straightforward. We have our user model that holds the list of playlists he has purchased. Then our playlist model, that is just a collection of songs. Then the song model, that belongs to an album that is from an artist. Then we have this PvP model that represent the link between the user and the opponent. This holds the current state of the game and the history of the interaction between the two users. In Google App Engine, this translates directly into the code. For instance, we have to write a Playlist class with all the properties we want on it and the type we want for these properties. And that's all you need to do. Then App Engine will create the necessary entries into the Datastore as soon as you start using this class. I would like to go further and show what is the real code you actually need to write on App Engine from when the request hits your back end to when you can deliver the response.
To do that, we can do it with a real example from "SongPop" that we call user/data. This handler is used to keep the device up to date, so it's called every time the user opens the application or goes back to this main screen. And we have to pull all the active games he has, so it can be up to 80 games, and this is called many times for each user session. So that's a critical call for us. If it fails, the user can no longer see what he can play. So what we really have to do is we query for the games, that is, all PvP models. And for each PvP, we have to retrace the playlists linked to the current quiz and the user model and his opponent profile. We structure it this way. So first, we fetch the current user, then we query for the list of games. Then we can extract the playlist we need from the games and the user and get the opponent's data.
Finally, we combine and format everything and send that back to the client. So I will show you, for each of these steps, how we implemented them and why is it efficient. So let's start with getting the current user. So we just extract the User ID from the incoming requests, and then we use a Python NDB API to retrieve the user model. So by just coding User.get_by_id. Why is this efficient? Well first, it goes through the Cloud Datastore that is designed to be highly scalable, in fact, it's built on top of BigTable. That is what Google uses internally for some of their products. But what really makes this line efficient is that it goes through a Memcache layer. So when we call our user/data handler again, it will directly fetch the user model from Memcache, saving us lots of time and Datastore operations. Also a nice feature with NDB is that the model is added to the instance memory. So when you have other parts of your code that also want to retrieve this user by this ID, you will directly get the model. You don't have to manage all of that. Now that we have the current user, we can query for the list of games.
So here is how we write a query using the NDB API. You can just do that and it works, but that would not be very efficient, and that would be expensive because this doesn't make use of Memcache. Indeed, NDB can't guess when each was invalidated, so it's up to you to implement this, but that's still very easy. We just define our own Memcache key. Look it up in Memcache. If it's not there, re-perform a query and set it into Memcache. Now that we have the list of games, we have the keys of the playlist we need, the keys of the opponents we want, so we can just group all of these keys and perform one single call to the Datastore to retrieve all of the models. Finally, we just have to process all of this data and move it into JSON to what we want to send to the client.
I will just spare you this code. So that's it. We implemented a request handler, and here is what real production code looks like. It's about 40 lines of code to implement this and optimize this. And this has run billions of times serving over 80 million users. So once we have this code ready, all we have to do is hit Deploy and let App Engine do its work. It will create the necessary entries into the Datastore, it will spawn the necessary instances as your traffic grows. So just sit and watch your app scale effortlessly. Here are a few more things I would like to share, especially with App Engine developers. One of these is make use of Memcache wisely because you will be allocated more space as your app grows, but that's still limited. So if you put everything in it, you will find that it's no longer efficient when you reach really high traffic limits. One other thing is that you should really only index what you really need to query on your models.
By default, everything is indexed, and that's very convenient. But if your models are often updated, that will be really expensive, and that will significantly increase your bill. That's just an easy thing to do, unindex what you don't need to query. Then don't worry about documented quotas on the limit about App Engine, because they are just there to make sure one single app doesn't abuse any resources. But actually, for "SongPop," most of them have been increased more than tenfold for us. So App Engine can really scale more than the limits that are documented. Finally, we were pleasantly surprised with the support we got with the Premier Accounts. You get to talk with qualified people that are responsive, and they really give you useful insights about what is going on in your application. So that was for the technical part. Olivier can continue. OLIVIER MICHON: So to conclude, I would like to say that App Engine really allowed us to scale. Scale to us, it's not really about not having to worry about hardware.
We expect a lot more. When we worked on "SongPop," we worked less than three months on it with our own two developers. We released it on iOS, Android, and the web, so we were on a very tight schedule, and we had no idea if it was going to succeed or not. But scaling means that it is very important to us that we write a line of code the same way if it's going to get executed 10 times or 10 billion times, like Alexis showed you. Because once we release "SongPop," it's when things really start for us. We're not shipping a box and move on. We are shipping a first version of our game, but it's more of a service. And actually, most of the efforts we put into SongPop were after the launch. So it's very important, if you want to keep your audience engaged, to release features and features.
Alexis here wrote most of the back end code all by himself, and when we started scaling to five million users, he was not worried about [INAUDIBLE] the database or things like that. He was worried about the new features that we wanted to implement to really keep the momentum of our game. So this is what scaling means to us, and I think it's very important for small teams like us. Now I would like to mention something about cost, because I think a lot of developers can be worried about cost on App Engine because it has a very different structure compared to what you see elsewhere. But I want to tell you that even for a game like us with more than 80 million people, and we give the game for free to all of our users, we still manage to make a profit. App Engine is not a prohibitive cost at all. In terms of cost to us, it's more of the cost of the missed opportunities. What would have happened if we had crashed when we were growing? What would have happened if we took six months or 10 months to deliver a full version of the game? So this is really what cost means to us.
I would like to say one last thing, one quick example about scaling with App Engine with a little example. Every week in "SongPop," we reset the score because we want to give you a chance, if you lost against your friend the previous week, to win the next week. So what happens is during a period of time of around five minutes, we have tens of millions of push notifications that are sent to our users to re-engage them. It's a great and important way for us to re-engage them at this point. But at the same time, what it does, it's going to have a huge impact on our traffic. And for less than five minutes, we go from, let's say, thousands of queries per second to more than 10,000 query per second on App Engine. So it's very important to us that during this time when we are re-engaging all of our users and preparing them for the next week so they have a good user experience.
For that, App Engine is perfect because it autoscales and it autoscales very fast, which means that it pumps new instances very fast because they are so lightweight. So we are pretty confident to say is that App Engine was very instrumental in the success of "SongPop," from fast prototyping to scale to 8 million users. Last slide. We are all day in the Sandbox with Alexis, so it's just over there. Don't hesitate to stop by. We are very happy to share our experience and more details about what we are doing. Of course, we can take also questions after this presentation. And we have more projects coming, so we are working on more things in the next few weeks and few months, and they are all going to be powered with Google Cloud Platform. And that's it. I'll let Chris talk. Thanks. [APPLAUSE] CHRIS ELLIOTT: It's absolutely amazing talking to developers and hearing these same kind of stories again and again. And the fact that it was mainly one person scaling to code that executes 11 billion times without any major issues, that's absolutely insane and really highlights the benefits of the platform.
And it was funny how you guys threw in the, we are hiring note, because we see such a huge demand for App Engine developers. If you're a qualified App Engine developer, please give us your resume because we have tons of people that keep asking us time and time and time again. So it's great. And that saying, code, deploy, and enjoy. When I first read it, that's an absolutely amazing way of phrasing it. And so if you see that at next I/O as more an official keynote or anything, you'll be firsthand witnesses to where the origin of that saying comes from. So last is our final reference architecture, and this is really dealing with dedicated game servers on the platform and harnessing the power of Compute Engine. And I play mobile games all day long.
I realize my boss is here, but I do play games all day long. Luckily, it's part of my job. But when I get home, I sit down, I boot up my gaming PC, and I jump in for some real time, competitive, player versus player, 64 people on a battlefield, all just going crazy. And that is really the other side of gaming that you see a lot of people having a blast with, and that is directly what we're targeting with this reference architecture. And so it's creating a full featured game experience that has all the functionality you expect, and once again, being scalable in a way that makes a lot of sense. So with that, once again, we created a sample application, and this is the sample application we put together. You may recognize GRITS from last I/O if you were here or watched any of the talks, and it's essentially an application to highlight HTML5 player versus player NoJS. And we took this core outline, and then we really applied a lot of the scalability concepts we're going to talk to, to the application to show you exactly what it looks like. And so we'll jump over to a demo real quick here. So this is what it looks like.
You basically log in, and you're hitting App Engine, and you can see a bunch of game servers that are allocated and how many people are playing. From there, you can directly jump in and do a quick match, or you can find a game with a reasonable number of players. So if I'm brand new, I'm probably going to want to play in a small server just I can pick up the game. And from there, you'll notice we're actually no longer on App Engine. We're actually at the IP address of Compute Engine. Obviously, there's lots of ways of doing it, but this is just a very verbose way of showing you what's actually happening under the covers. From there, you jump in, start a quick game, and all of the assets will be pulled down from the server– so the media, the music, all of the graphics– onto the web browser client here. And while that's happening– it usually takes a little bit longer up here on stage on the Wi-Fi– we also have an admin console that you can log into and see here.
And this allows you to start up a cluster, spin down a cluster, and it's also a great place to go to experiment with autoscaling, which we actually implemented in here as well. So after the average load on the server passes 80%, you'll end up adding new servers automatically and spinning up to handle the number of players who keep joining the game. So if we jump back, we should be in. And basically, we're in. So you respond, and it's fairly basic. It's highlighting some cutting edge web technologies. I mean, it's developed by Google. We don't have a massive team of AAA developers. But it kind of highlights running around, real time motion, shooting, et cetera. A bunch of people can join in and play. And it's always really funny after showing this in a presentation, especially at Google, usually I look in afterwards and there's 50 people playing, even though there's somebody else up presenting afterwards. It's kind of fun to see people get in and really have fun with the application.
And so jumping back to the slides, all of this source code is actually open sourced as of two days ago, so you can go in, pick it up, start playing with it, and get a really quick way to start scaling on the platform with dedicated game servers. And swap in whatever you want to replace the game assets that we provide. So with the architecture, the big difference here when you look at the devices is it's no longer mobile devices. You're looking at everything from gaming PCs, to web browser. It's up to you as a game developer to really use what technology makes sense for the game you're developing and the players that will be using it. From that point on to the magic of Cloud Endpoints, really connecting to Server Matchmaking, and this can be as simple or complex as you want. It could be, I want to play Capture the Flag with people of reasonable skill nearby, and have all the matchmaking logic to make that happen and return a single IP address to the client. Otherwise, you could do things of just give a large list of IP addresses in a region and allow the client to go in, see which one has a lower latency, and join whatever map they choose to play.
That's just one of the many ways of building it out, and the key aspect is App Engine has a scalable component to really scale up and provide the technology to power whatever type of matchmaking algorithm you choose to implement. From there, you saw on GRITS what we looked at, very straightforward. List of IP addresses and also just Quick Match. And so from there, this is really where the magic happens and where a game developer should spend the majority of their time, is really focusing on creating the core game play experience that keeps people riveted on the edge of their seats. And so that's really connecting your game clients to the over full duplex sockets to the game engines running on Compute Engine, and having that real time gaming, low latency, all the benefits of the platform there are really what power these type of applications. And what's really interesting here as well is as you start to grow, you can increase capacity in different regions and you're not necessarily tied to being overprovisioned in an area where it turns out your game is not as popular.
Next up, going back to App Engine, a lot of cases you have in game requests. So it could be something as simple as at the end of the match, you have a bunch of players, and you want to keep track of their scores, kill death ratios, et cetera, long term. That's a great place to go, post it to App Engine. That can all scale up and keep track of all of the user profiles there. Otherwise, if you're doing things like a marketplace, it's a great place to go back and verify that somebody has purchased the plus five sword of greatness. All of the infrastructure, once again, can support those full featured functionality you'd expect for in game requests. Up next, the final component of the App Engine implementation here for the Server Matchmaking is really the virtual machine orchestration, so really scaling up and scaling down as your player base grows, and scaling in a way that makes sense. So you can even do some more complex logic that, for example, if you notice it's nighttime, all the gamers have gone to sleep at 5:00 AM and have a bunch of excess capacity, it's very straightforward to then remove those servers from matchmaking and terminate them and bring them back when the players come back next night.
And with the sub-hour billing that we announced yesterday as well, it makes it even easier to start to build out these autoscaling algorithms because you're not worried about having an instance provisioned for an hour and being charged for an hour. You're only charged for the minutes that it's actually up and running, which makes this a lot easier to implement and reduce the complexity there. So we saw this on GRITS, what it looked like, and it's very straightforward. This actually, Admin Console allows you to play with your autoscaling algorithms as well. So you can go in, jump, set your load, and then watch the autoscaling algorithms take place and scale up the number of instances running. And finally, the last bit really is that MapReduce dynamic duo between MapReduce and BigQuery, and using this powerful technology to get a lot of insights into what your players are doing. And for those of you who have been taking pictures the entire time, I feel bad, but this is the final slide. I'm not going to add any more components just to keep trolling everyone.
And with this, if you're running game servers and you have tens of thousands of instances, which a lot of these games are scaling up to, you have a huge amount of game logs, and that's a great place to put them in cloud storage and then run a lot more complex processing over them to get those insights you really need as a game developer to be successful. And I kind of lied, actually. Glass integration, once again, think about all of these components, all this cool game play that is possible. It works perfectly on the platform as well. And actually with Glass, if anybody in the room creates a Brick Breaker application, please let me know. I've been dying to have Brick Breaker because that would actually work perfectly on the [INAUDIBLE] scrolling forward and backwards. So let me know if you come up with that. So that really is just the matchmaking aspect. And really, it's about creating a full featured game, keeping all of the user's information around, all of their stats, et cetera, to having friends, to creating a marketplace, to creating events, social engagement, all of these features that games really need to be successful long term, App Engine is a great place to build out and augment the functionality.
Just like with FreshPlanet, they launched, and then they could quickly iterate, add new features, and really keep people engaged. Those sort of concepts also apply to this type of dedicated server infrastructure. And so to wrap up, time and time again, we see that from small indie developers to AAA game studios, the Google Cloud Platform is the place for cutting edge developers to build out these really innovative experiences, and we continue to see people really pushing the boundaries and blowing us away with what they're building on the platform. And the Solution Papers for the mobile architecture, the iOS Push Notification, and the Dedicated Game Server are already published or will be published very soon to the Cloud Platform website. If you go to the website, there should be Resources button. You can click there, and all the papers will show up. Otherwise, keep an eye on the newsletter for a lot of the latest up to date information there. Otherwise, the sample applications are all available on GitHub.
So the GRITS one is live today. The Mobile Backend Starter is ready there as well. And the mobile gaming one, the riddle game, if you really want to know what the last few riddles are, should be published pretty soon, and we'll announce that in the newsletter as well. And there's other relevant I/O sessions as well. I've gone through and talked to the teams building these out. Today later on, there's one on mobile, From Nothing to Nirvana. And it's a really great example of how is very easy to get up and running and start building out an application that harnesses the power of App Engine to really create a scalable game. Otherwise tomorrow, the OAuth talk is a great way to see how all of these components really fit together and solve complex problems, and it is very seamless. This is the talk that justifies how many times I've been using the word "seamless.
" And also, the 10:00 AM talk is really cool. It's with Game Sys, and they talk about building out a massive scalable infrastructure and how they really harness the power of BigQuery to get some really cool insights. It's a really good talk. And yesterday, JAM with Chrome had a talk. Really cool, since we had a quote in the presentation, highlights one of those cool things that people are building out. And even the slot car racer that you saw in the keynote as well, that's Compute Engine, App Engine, really highlighting that dedicated server aspect. You could talk to Ed as well if you'd like more information about that. And that's really it. I'd like to invite Alexis and Olivier back up to the stage, and we'll take any questions that you have. There's microphones on either side, so please line up. Otherwise, please fill out the survey when you leave and rate, and we'll keep you up to date on the latest status of the riddles. Yeah, question.
[APPLAUSE] AUDIENCE: Hi. I saw you were using cloud storage BigQuery to store, I guess, all the game information. But why not use Google SQL? Is it because [INAUDIBLE] scale the same, in which case, we need to use either of them? CHRIS ELLIOTT: Yeah. So the question is, why aren't you using Cloud SQL for some of this analysis and insight? And that's actually another perfectly good option that you can put in there if you really are familiar with using SQL, you need a database that scales and manages. And if your data sets are reasonable size, that's great. The reason we put BigQuery in here is really, you're looking at billions and billions and billions of rows of data. And once you start to put that size of data in a managed service like Cloud SQL, it's really going to start to break down because SQL is just not designed for that kind of data processing. Whereas if you read the Dremel white paper, it's amazing technology we have that's really making a possible. AUDIENCE: So basically, it would be just a scaling issue? CHRIS ELLIOTT: Yeah. It's really being able to get a lot of big data insight. I mean, MySQL and big data usually don't mix frequently.
AUDIENCE: Thank you. CHRIS ELLIOTT: Yeah, no problem. Yeah? AUDIENCE: Hi. I'm Simon [? Peter ?] from Quebec City in Canada. CHRIS ELLIOTT: Cool. I'm a Canadian as well. AUDIENCE: Nice. My question is for the "SongPop" application. How do you manage to get "SongPop" profitable, even though you make very heavy usage of the App Engine platform? OLIVIER MICHON: So we have different ways we monetize. Again, we have some advertising on the free version, we have in-app purchase, and we have these kind of different things. I think it's a basic business model. But my point was that even with just an average monetization for an app, App Engine cost is not prohibitive. You still manage to be profitable for an app that you essentially distribute free for everyone and just put advertising in it. AUDIENCE: Thank you. CHRIS ELLIOTT: Cool. Nobody asked about Google Play Services? I even created a slide right after this one for a segue. Should have planted a question. Well, I'll show you the slide anyways. So if you're thinking about Google Play as well, how that fits in, it's really kind of looking at these dimensions.
Flexibility, you have increased flexibility as you start to use App Engine and Compute Engine, you have more and more freedom to do anything you want. But it's much easier to develop with something like Play if you're just a front end mobile developer and quickly want a leaderboard or stats. So the key option is you have the flexibility to implement anything you want on the platform. And it's up to you as a developer to choose what tools best fit your use case, but we have all of those tools to enable those next generation experiences. So thank you for asking that question. All right. Well thank you, everybody. [APPLAUSE].