Video: App Engine Hangout – chat with an App Engine Software Engineer in Test
AMY UNRUH: Hi, everyone. Welcome to an App Engine Hangout. So this week, we have with us Robert, who is an App Engine Test Engineer, Software Engineer in Test. And this is really a great opportunity to– he's going to talk about some of the test beds and test runners that he uses, and that you can use as well. And he's going to talk a little bit about what he does too. So this is a really great window into the job of one of the people who helps App Engine work as great as it does. And so I just want to do a little plug before we switch to the studio for what's coming up next week. We're going to have a chat with one of our big App Engine customers.
And we'll announce a little bit more about that a little bit later in the week. OK, and I want to apologize. We said earlier that we could invite some people who might be interested in joining the Hangout just to chat with us. But we had some technical issues. Sorry, we got started a little late. And so we're unfortunately not able to do that this time, really wanted to. And we hope to be able to do it in future Hangouts. OK, so let's switch to the studio, where we have Robert and Danny. And what I'd like to do is this just to get started, just ask Robert a few questions about what he does, just because I think it'd be of interest to everyone watching. DANNY HERMES: Hopefully, you don't catch him off guard, huh? AMY UNRUH: Yeah. No, this is this not a big grilling. But I think everyone would be interested, before we start with what you actually have planned to demo, just tell us a little bit about what you do in your day-to-day work. And maybe how a little bit about how you got started with it, too, if you think people would like to hear that. ROBERT SCHUPPENIES: OK, so I joined Google and App Engine three and a half years ago as an SET, Software Engineer in Test.
And my job is basically to write software that helps with testing for the three major user groups. For one, App Engine developers– so people who write App Engine, to help them make sure App Engine doesn't break when we push a new release, for example– then the external developers, like everyone who's watching, who write applications that run on App Engine. And we want to provide tools so that when you write your application, you can test them easily. And the third group are internal developers at Google who also write App Engine applications. And they also want to make sure that the applications they write are testable. And because Google has some special infrastructure, I also work on making this infrastructure usable for tests for App Engine. DANNY HERMES: So are you a polyglot in your day-to-day life? Are you targeting every runtime? Or is it just one specific runtime? ROBERT SCHUPPENIES: Mostly Python.
DANNY HERMES: Mostly Python. OK. I knew today, we were mostly talked about Python. But I wasn't sure in your day job in Test, if you're targeting all the runtimes, or if you have specialists for each individual one. DANNY HERMES: So we split it up. We have other Software Engineers in Test. Another colleague, Dave [INAUDIBLE], he writes on Java, works in Java. ROBERT SCHUPPENIES: Sure. Cool. Back to Amy. You have some more questions, I'm sure. AMY UNRUH: Well, I didn't want to subject Robert to a big grilling. So actually, let's just start with what he had planned to demo. And just for everyone watching, if we have a little bit of time at the end, we'll go to the moderator queue. So we hope to answer just a few of your general questions. And if we have a bit of extra time at the end, we can also sort of make this a bit of a general Office Hours, as well.
ROBERT SCHUPPENIES: So what I want to walk through today are three test tools we use and also have available for external developers. And one of them is for unit testing. Then I want to talk about handler testing, which was also a question from the page, the Moderator page. And then I want to talk about end-to-end testing, which was also a question. How can you test your whole stack? And finally, if we have time, talk about library testing. If you write libraries for App Engine, how can you test that? DANNY HERMES: Right. ROBERT SCHUPPENIES: So just go ahead and do it? DANNY HERMES: Yeah. ROBERT SCHUPPENIES: Can you see the screen? Is it big enough? DANNY HERMES: You probably want to pump the size up by one or two. ROBERT SCHUPPENIES: OK. AMY UNRUH: You might want it a bit bigger. DANNY HERMES: OK. DANNY HERMES: Maybe a couple more. ROBERT SCHUPPENIES: Wow, that's going to be– DANNY HERMES: It will be. But it's more than 80 characters, right? ROBERT SCHUPPENIES: Yeah, let's see.
DANNY HERMES: You can reset your PS1 to something shorter, export PS1 equals. ROBERT SCHUPPENIES: I'm going to leave it now. Otherwise, I mess up my bash and then maybe not even– DANNY HERMES: It's just PS1. You can just close it and restart bash. PS1 equals and– you don't need the dollar sign. Well, you don't want that. ROBERT SCHUPPENIES: So what do I want? DANNY HERMES: PS1 equals say, demo and then dollar sign and then a space. ROBERT SCHUPPENIES: OK, this one died. Export PS1 equals– like this? DANNY HERMES: Well, put it in quotes and put a space after it. Yeah, like that. It should be fine. There you go, a lot more space, right? ROBERT SCHUPPENIES: Now I'm getting lost in my directories, but that's– OK, so what I want to do– DANNY HERMES: Whatever. ROBERT SCHUPPENIES: Three things.
The first thing– I think I can increase that, too– is the testbed. Testbed is a library we have for unit testing and Python in particular. There's also an equivalent for Java that helps you test your Java applications on a unit test level. And unit testing is important because software, if it's not trivial, it's made up of lots of pieces. And you want to test those individual pieces because if you can identify the piece, the particular piece that failed, it's easier for you to debug it and fix it. So unit testing. And for Python, it's called testbed. For Java, I don't even know if it has an explicit name. But you can always just do a Google search– or whatever your favorite search engine is– App Engine and then local unit test. And that will either bring you to the Python or to the Java site.
And I'm going to talk about Python today. And all the information I'm going to talk about, you will also find on this page. So let's see. For Python, since I don't see the path I'm in– DANNY HERMES: PWD will tell you the path, if you want it. ROBERT SCHUPPENIES: OK, testbed. So I just set up a demo here. And I will use a run script to run the tests. And the run script is also available on the page I just showed. So you can just download it and use it. DANNY HERMES: And what are you going to be testing? Just the basic hello world app or something else? ROBERT SCHUPPENIES: No, we're just going to test the units of an app. So we're not going to test an actual app, but only pieces. And for App Engine, when you write an application, you will use services of App Engine. You will use Datastore, Memcache, [INAUDIBLE], things like that. And often for unit tests you want to mock that.
DANNY HERMES: Right, using stubs or some other local mock. ROBERT SCHUPPENIES: Right. And for App Engine, we now have this test library that provides those stubs. DANNY HERMES: So the test runner– are you going to talk about the app that you're running it on? Or are you just going to run it and talk about how it's– ROBERT SCHUPPENIES: This is just going to run in the shell. And I'm going to show you the tests we have and then how to run it, and what the dependencies are, and how it looks. DANNY HERMES: Cool. ROBERT SCHUPPENIES: OK, so as I mentioned, we have the test script, which is just this one. Let me set up the path, PYTHONPATH equals– and I'll put this. The test runner, what it does, it uses unittest2, which is a Python library, which is similar to unittest, which is in the standard library. But it has extra features like finding tests in directories. So what you can do is you say, give me all the tests in this directory.
And then it will just go through the directory and find all the tests. So that's why I use unittest2. And as you can see here, it expects two input values in the SDK panel. So the SDK, the Python App Engine SDK, you want to use and the root of the test bed. DANNY HERMES: Which is probably just current directory here. ROBERT SCHUPPENIES: Which is, yeah– and as I mentioned, we have a couple of dependencies, which for this case, are unittest2. I already mentioned that. And then webtest. I'm going to talk about webtest later. But it's important if you want to test handlers. Handlers are the pieces that actually render the website and serve it to the user. So this is also something you want to make sure that actually works with the test. So those are the two dependencies we have. And we have tests in the tests directory. Here we have three tests– Datastore, logs, and handlers for modules for three types of tests. And as an example, I'm just going to look through the Datastore. DANNY HERMES: Emacs, man after my own heart. ROBERT SCHUPPENIES: There's a– AMY UNRUH: Me, too.
ROBERT SCHUPPENIES: –small followed by– DANNY HERMES: All three of us, Emacs users, wonderful. ROBERT SCHUPPENIES: OK, so what are we going to do? First, we input unittests. We say, hey, we want to use the unittest mode, because want unittest. Then we import the modules. We're going to use the App Engine services– in this case, Memcache, database, db, and then testbed, which allows us to use the [? steps ?]. And testbed comes from the ext package. In this particular case, we have a TestModel, which is just a simple model that has an IntegerProperty, text property to hold some data. And we also have a function here that's called GetEntityViaMemcache that if you retrieve an entity through that function it will first look it up in Memcache. If it's not there– DANNY HERMES: So in other words, what NDB does by default. ROBERT SCHUPPENIES: Yes.
Now that we have NDB, we should use NDB. But it's a good example for this. DANNY HERMES: Yeah, definitely. Except there's one level of caching not even included there, which is the instance-level cache. But that's a conversation for another day. That's what we're testing here. ROBERT SCHUPPENIES: But NDB is a nice library. [? Gigoed ?] is best. So what do we do? We write a unittest, the usual unittest or test case as a base class. And then we set up our test. So default thing is you say, hey, setUp– this is what is done before every single test you do. And the counterpart for this is tearDown, which is done after every single test you execute. DANNY HERMES: So I notice you have in your setUp, init_datastore_v3_stub. Can talk a little bit about what that v3 is doing? And how people can know what stubs and what stub names correspond to the services they're going to use? If you're going to talk about this later, by all means, hold off until then.
ROBERT SCHUPPENIES: I'm just going to– we are two steps away from that. DANNY HERMES: Oh, OK. Cool. We'll wait til then. We'll wait two steps. My bad. ROBERT SCHUPPENIES: So first we instantiate a Testbed object, which is an object that will create the stubs for you and then mock out certain capabilities so you can use them for testing. Once you've done that, you say, hey, activate the Testbed for me. This is the step where you say, whatever has been there before, save it, so when we deactivate it later on, you can restore it. And then we say init stub. And we have different services. Each service has a name. And then it's init service name stub. So init_memcache_stub's obvious. Then there's init_taskqueue_stub, init_ [? make ?] stub, and then datastore_v3_stub, which is not that obvious. Why is it v3, right? So that's– DANNY HERMES: And was there a v2, et cetera? ROBERT SCHUPPENIES: Yeah, it's just an iteration of the Datastore versions we had.
And this is v3. I'm sure at some point there will be a v4. Because we don't want to have you deal with regressions when we switch to a new dadaists. We'd rather be explicit– rather be explicit than implicit. DANNY HERMES: So I actually know a bit of the history behind the naming there. I was just– for the viewers' sake. But the question more that I had to ask was how would we know that from google.appengine.api.importdb would correspond to a stub that's Datastore? Is there any explicit way to do this? Or is it just knowing what's there? ROBERT SCHUPPENIES: Two things you can do. You either look it up on the website, which says, hey, here are all the supported stubs. Or if you just say, whatever, I don't care init_all_stubs. DANNY HERMES: Oh, OK.
ROBERT SCHUPPENIES: OK. So that's– DANNY HERMES: But that will make the test a little slower and certainly less explicit, right? ROBERT SCHUPPENIES: Yeah. DANNY HERMES: OK. ROBERT SCHUPPENIES: So you can do both. OK, restored. Now the first test is really straightforward. We just create an entity, put it into Datastore, and then try to retrieve two entities. But the test says, hey, check that it's only one entity. And now that we use Testbed and activate it before every test run, tearDown after every test run, we will work on an empty Datastore. So this test will actually pass because no matter how many times you run it, you tear down the Datastore, create a new one. So that's what it does. The next test is really just checking hey, create a key, delete the entity, see if when I retrieve that key, I don't get anything returned. And this also already uses the GetEntityViaMemcache function, which just looks into Datastore, and if it finds it– OK.
Then I'm just going to skip here. This one [INAUDIBLE] entity then retrieves the entity via Memcache. And we want to check that's actually the entity returned we have created. So it's just again testing the path for that function. DANNY HERMES: Sorry, I don't want to hijack– ROBERT SCHUPPENIES: No, go ahead. DANNY HERMES: So I notice one thing, which made me think. We have two runtimes supported in Python– 2.5 and 2.7. And unittest actually changed between– well, it changes every version of Python. But there was initially assertEqual, assertNotEqual, assertTrue, assertFalse, and that was about it. But in 2.7, there were like six more asserts added, one of which is assertNone. Now I see you're doing an assertEqual with a none. So is this written with both runtimes in mind? Or are there separate test cases or separate ways you're going to write it for the different runtimes? ROBERT SCHUPPENIES: So this is really up to the app developer.
If he says, my app will only run on Python 2.7 because it's the future, and Python 2.5 is going to go away at some point, you can use the test functionality that is available in Python 2.7– so assertNone and all those extra fancy features. DANNY HERMES: But Testbed itself will run on both. ROBERT SCHUPPENIES: Is agnostic, yeah. It really just uses– it's really just a utility. It doesn't depend on unittest. It will just provide additional tools for your unit testing [INAUDIBLE]. DANNY HERMES: Like making these stubs. Cool. Sorry. ROBERT SCHUPPENIES: No, that's fine. Also, Python 2.7 actually includes unittest2, which I specifically included here as a library– DANNY HERMES: As part of the standard library. ROBERT SCHUPPENIES: Yes. DANNY HERMES: OK, great. ROBERT SCHUPPENIES: And then the last one, the last function I'm testing here, or the last piece of functionality, I create an entity then get the entity key, delete that entity, and see, if I use GetEntityViaMemcache, it's still available. Because GetEntityViaMemcache is not aware of Datastore deletes, but it cached it. It would find it in cache and then [? return ?] it.
DANNY HERMES: One of the two hardest problems in computer science, right– cache and [INAUDIBLE]. [LAUGHING] ROBERT SCHUPPENIES: Yeah, so this is a simple example. Don't use that in your application. DANNY HERMES: Please. ROBERT SCHUPPENIES: Please. DANNY HERMES: Pretty, pretty please. ROBERT SCHUPPENIES: Yeah, and so this is what's done. Now if we– where are we? We're here. If we use the run script and point it to the SDK, because that's one of the parameters and point it into the test directory, it would go through all the tests, execute them, and then this [INAUDIBLE] what we see here. DANNY HERMES: Great. Another side question– are you on the newest 1.7.3? Or are you using 1.7.2? ROBERT SCHUPPENIES: That's actually 1.7.2.
DANNY HERMES: Aw, come on. If people didn't see, 1.7.3 was just released yesterday. So if you haven't upgraded yet, go ahead and do it. But apparently we can't even get our own test engineers to do it. ROBERT SCHUPPENIES: Well, actually, I've worked already with 1.7.4 but I didn't want to use it. DANNY HERMES: Oh! ROBERT SCHUPPENIES: Better to be safe than sorry. AMY UNRUH: Robert? Just for fun, why don't you edit one of those tests to have an [? inclusion ?] break and show what it looks like, if that's the case? ROBERT SCHUPPENIES: So we can just introduce a simple test_fail– DANNY HERMES: Contrasting styles there. ROBERT SCHUPPENIES: What do we want? False. OK, so we now [INAUDIBLE] DANNY HERMES: There we go. ROBERT SCHUPPENIES: We fail. And here's what you get– so just failing test. And one thing we don't provide is, for this particular case, integration into a tool, like Eclipse or PyDev for Eclipse. Because often, people have different sets of tools and just to support all of them is really hard.
So we just give you the basics and then you can hook it up. DANNY HERMES: And, I mean, as far as what we're talking about today, the Python community isn't necessarily married to one or a few set of tool [? chains ?]. There's– ROBERT SCHUPPENIES: [INAUDIBLE] tests, for example. DANNY HERMES: Right. So we live in harmony in that way. ROBERT SCHUPPENIES: Another thing I wanted to talk about, and there was also a question– how can you test webapp2? Webapp2 is a library, again, that runs on App Engine and helps you to render HTTP response. DANNY HERMES: And create WSGI handlers and things. ROBERT SCHUPPENIES: Yes. So let's have a look at this. What we have– a simple handler here that uses webapp2 and does nothing other than writing header word and setting the content type. So it's really simple. If you run that in your app– DANNY HERMES: It's the Python 2.
7 hello, world. ROBERT SCHUPPENIES: Hello, world. And what you also see here, we use testbed because we want to test it. And that's actually– ignore that. DANNY HERMES: OK. ROBERT SCHUPPENIES: This is the test part. We're going to use webtest. And the test itself again is just a [? check ?] class of test case. And then in the setUp, we first create the app, the WSGI app, we're going to test, which is the handler we have and then some path that works. And then we use webtest2. Webtest2 is a great library for testing web applications for Python. And it really makes sense to use that in the App Engine context as well. DANNY HERMES: Is this a best practice for our Python developers for their local testing? Or is it just something that we are big fans of? ROBERT SCHUPPENIES: You mean using webtest? We are a big fan of it. DANNY HERMES: OK. We're not going to make you use it. But– OK.
ROBERT SCHUPPENIES: It makes it really easy. DANNY HERMES: So is there some reason the import of webtest happened after the definition of the class? Or that's just– ROBERT SCHUPPENIES: Oh, this is really just– so it's– DANNY HERMES: Just thrown together. ROBERT SCHUPPENIES: –focus. DANNY HERMES: Gotcha. OK, great. ROBERT SCHUPPENIES: So you know what– DANNY HERMES: So it's not that it's just thrown together. It's important that you draw your eye to. ROBERT SCHUPPENIES: It was intentional. DANNY HERMES: Got it. That's why you broke PEP8, huh? ROBERT SCHUPPENIES: Yeah. And I don't run Pylot here. So what we do now– DANNY HERMES: We don't run Pylot here. All right. ROBERT SCHUPPENIES: On this demo code. We do it at Google. DANNY HERMES: Yes. ROBERT SCHUPPENIES: OK, so now that we have– DANNY HERMES: So [INAUDIBLE] your code everyone.
ROBERT SCHUPPENIES: Now that we have the app, we're going to wrap it in a test app that is provided by webtest. So we say, webtest or test app, please wrap this. And this test wrapper allows us to easily make requests like get posts– whatever we want. So in this example, we say testapp.get, which is making a get request to the app and will give us a response. The response object is also a webtest [? wrapper ?] object. And this one allows us to easily query this status code, the body, or the content type, for example. So the three things we want to check for our very simple handler. And that's basically it. So if we– we can also, for example, just copy this, and we'll make this a post request. And then just say hey, instead of get, do post.
So it works the same way. DANNY HERMES: Or we could set a header and check that, also? ROBERT SCHUPPENIES: You can do whatever you want. And something you often do on App Engine is use App Engine services, like you use Memcache, for example, when you want to retrieve your stuff faster. So how do we merge the two tests things here– the webtest and service testing? So the testbed thing we talked about earlier. Well, again, a simple handler that would just, on a post request, parse the key and the value from the URL parameters and put them into Memcache, [? some other thing ?] big here. And the test now uses testbed again, what we had before. And before we also used Memcache. And we just do the same thing we saw before. We create an app in the testbed wrapper. Then we instantiate testbed and initialize the Memcache stub. And for tearDown, we're just going to deactivate it. So what we do now in the test, we say this is the key, and this is the value. [INTERPOSING VOICES] And we're going to pass this to the handler. And then the test itself checks, hey, if I [? query ?] Memcache now, is the key set and does it return a value [? assigned ?]? So straightforward.
DANNY HERMES: So first we saw services only. Then we saw webtest only. Now we see both of them used at the same time, which is probably what most of your web apps are going to look like. ROBERT SCHUPPENIES: Yes, most web apps use some App Engine services. OK, yeah, that's pretty much what testbed allows you to do. So it allows you to write unit tests. And unit tests really– the greatest benefit is that they run fast, or that they should run fast. So you can run them whenever you save or whenever you commit code. And you get instant feedback, which is a good segue into the second type of test, end-to-end test, which allow large tests. They test the whole stack, from top to bottom. And because of that, they're really slow and they take a while to set up, and then also a while to execute. But it's so important to do end-to-end tests because while you may trust the units, you don't know whether the whole setup will work, whether the whole configuration, wiring you have done, actually is correct. DANNY HERMES: Sure.
ROBERT SCHUPPENIES: And part of App Engine for– AMY UNRUH: Robert? ROBERT SCHUPPENIES: Yes? AMY UNRUH: Sorry to derail you there. I thought before we moved on from testbed, one thing that might be of interest to a lot of people, because I see a lot of questions about this, is testing with HRD when you have issues of the eventual consistency. Would you be willing to say a few words of advice about that? ROBERT SCHUPPENIES: So like a particular issue I should talk about. So one thing I can mention, but you I think you know, on the– AMY UNRUH: People might know. There's a setting to indicate this. ROBERT SCHUPPENIES: If you go to the website, [INAUDIBLE] world, describes [? to ?] you exactly what you need to do. So here you say, hey, we're going to use the PseudoRandom HRD Consistency Policy. And if you enable that in your tests, your tests will actually work with HRD similar behavior.
So that is not consistent by the way. You have some certain random [INAUDIBLE]– DANNY HERMES: It stimulates the network latency, effectively. AMY UNRUH: And you can turn that on for some tests and turn it off for others. DANNY HERMES: Also people who are testing their apps just by running them and playing around with them by clicking, keep in mind, dev app server by default right now is still master-slave. So you're going to have to pass in a flag to actually make it be HRD when you're just running your development server. And that's also something that people don't always know, don't always realize. ROBERT SCHUPPENIES: Yeah, any other questions? OK, then I'm going to go back to the end-to-end tests– and hope I find where my directory is. So end-to-end tests– as I mentioned, you want to test the whole stack. And to test the whole stack, you need to run the application in some way. An example is you have app.yaml configured and say this handler should work on this path, and this path should be treated by this handler. And this is something you cannot test in the unit test.
You want to check that your app.yaml configuration actually is working. So for this, you have to run your application. So one thing you can do is start your application with dev app server and start clicking. DANNY HERMES: Sure. ROBERT SCHUPPENIES: Hey, is this thing working? Which is– DANNY HERMES: it's hard to automate your finger. ROBERT SCHUPPENIES: Insidious, right? You can have one of those click– I don't know what they're called– sipping birds, right? DANNY HERMES: You can also use something like Selenium. ROBERT SCHUPPENIES: You could use something like Selenium web driver. But you still need something that starts up your app. And in order to automate that, there's a library called gaedriver. And gaedriver basically does exactly that. It takes an application and some other configuration input and says, now based on that, we'll start your app with dev app server, or we'll deploy it to App Engine. And this can be done with an API.
So it can be part of your test setup. So you say, hey, please deploy this app and then run the following tests. And this is what I'm going to demonstrate now. Let me see, I need to set up the path again. So let's look at the app at first. The app is just the standard app.yaml main. And it also has a static directory, which just has a static file, nothing special. And then what we also have in here is the configuration file we're going to use for this test run, main. And this one includes the app ID. So if you want to deploy, the library needs to know where to put it. And it also needs an SDK so it can actually deploy it. It also knows where the application is located. And in this case, cluster host name is set to local host, which the library will take as, OK, use dev app server to run the tests. If you use appspot.
com, it will deploy the app to appspot.com. And in that case, you would also need credentials. DANNY HERMES: And I'm assuming that it's going to parse the port from this and pass it into dev app server– ROBERT SCHUPPENIES: Yes. DANNY HERMES: –when it runs it? OK, cool. So let's look at the test module. Here we have, again, unit tests, some unit test library and then gaedriver, which library we use, and then the config file I just showed, which has the crucial config information. Two things we're going to do here. We will work with an app token, which is something gaedriver will give you to shut down the app again. For example, when you started with dev app server, it's started. But you need some way to tear it down, otherwise the process keeps running.
So you need to close it appropriately. And then a config object, which is basically the config file parsed and then given to you as an object so you can use it in– DANNY HERMES: The config file in app– excuse me, the config file in app.yaml? ROBERT SCHUPPENIES: The gaedriver config file we just [INAUDIBLE]. DANNY HERMES: Oh, OK. ROBERT SCHUPPENIES: The app ID– DANNY HERMES: OK. Gotcha. ROBERT SCHUPPENIES: And cluster hose name and things like that. Because enter-enters are expensive, you only want to do the setup once, or as few times as possible, as necessary. And for this we use setUpClass, which tells the unittest runner only do that once per testclass. So if we have five test [? members ?] in the class, it will still only do it once. And we're saying for tearDown class. So what are we do here? At first, we retrieve the config object. So we read basically the config file and then have an object that contains all the information.
DANNY HERMES: This setUpClass method, this is not part of unittest. This is some extension of unittest– ROBERT SCHUPPENIES: No, it's actually unittest. DANNY HERMES: Wow, great. So what you're doing, you're making a custom test case. And unittest– well, unittest.TestCase and unittest2.TestCase both support this, having a class method called setUpClass? ROBERT SCHUPPENIES: Unittest2 supports it. Since unittest2 is part of Python 2.7, you can assume Python supports it. DANNY HERMES: Right, right, right. ROBERT SCHUPPENIES: OK? Good. So in the second step, in the setUp classes, just gaedriver setUp app, and based on the cluster host name, which we have set to local host, it will start the app dev app server and then return to the control flow. And that's pretty much all you need to do to set up your application for testing. And then let's look at one of the tests here, test_homepage, where you build the URL from the config object, in this case config.
app_hostname, would tell you, OK, local host port 8080. For when you deploy your app, it would be the foo.appspot.com So you build your URL, then you make a request to it, and you just check the response, whether it's what you expected. And if we look at the application, hello world, and this is what we want to see, and the same for dynamic content. Oh and then, sorry, forgot to mention that this test also checks ETag headers. ETag headers are HTTP headers that are set to indicate to a client, cache this content or don't cache it. And if it has an ETag header, it will actually cache it. And for dynamic content, like what is generated with a handler, App Engine will not send an ETag header, because dynamic content is generated dynamically, so you don't want anyone to cache it.
On the other hand, if you have static content like whatever you serve– DANNY HERMES: Static.txt ROBERT SCHUPPENIES: Yes. App Engine will set an ETag header for you. And then a browser queries it once, fetches it once. And for the second request, it would just reshow the same file. And here we just check that. We build the URL again, make a request, and check whether the ETag header's in the response and the same for the static content where we actually expect it. And let's run it. DANNY HERMES: Now you said it's going to take a long time, yeah? ROBERT SCHUPPENIES: Well, let's see how long it takes. DANNY HERMES: Obviously, longer than 0.0051 seconds? ROBERT SCHUPPENIES: Definitely. Longer than the past. But we're done. So it took five seconds. Five seconds to start the app dev app server, make the request, and then tear down dev app server again.
For appspot.com it will take longer because deployments usually take longer. So that's end-to-end testing. And this is something we also use as part of the App Engine development, because one thing we want to test is that all the APIs work on the entire stack. So we have, for example, applications built from tests from Memcache. So we check, OK, if I use Memcache, does it go all the way through to the Memcache back-end, come back and have the correct response? So this is part of what we use internally, but then open source because it's useful to others as well. DANNY HERMES: Now, is there some equivalent search term in your favorite search engine for gaedriver, like there was for webtest? ROBERT SCHUPPENIES: The search term is basically gaedriver, which will bring you to a Google project code page. And if you want to install it on your machine, you can just do a pip install gaedriver– DANNY HERMES: From PyPI. ROBERT SCHUPPENIES: –which will fetch it.
OK. And yes, since this is an API, it's easy to integrate that into continuous test setUps you have. Because I saw a question on the– DANNY HERMES: Moderator– ROBERT SCHUPPENIES: –Moderator page. DANNY HERMES: –about continuous deployments and testing. ROBERT SCHUPPENIES: So you could use gaedriver for that to say, OK, deploy the app, run your tests, see if it works. DANNY HERMES: Sure. ROBERT SCHUPPENIES: OK. And then that brings me to the last part, which is library testing, where we also want to support developers who write libraries for App Engine that are then used for App Engine applications. and– DANNY HERMES: NDB is actually a good case for this, right, because it started outside and then made it in. But it's still an ext, so maybe– ROBERT SCHUPPENIES: But it's a nice development model, I think, because it allows you to get feedback quick from users, which is what you really want.
So let's see where we are. So for this, I'm going to use a library called aeta. Aeta also started out as an internal project. And this summer our intern, Jacob Taylor, actually polished it and made it an open source project. So again, you can just search it– aeta App Engine. DANNY HERMES: What's the acronym for it? Or is this something else? ROBERT SCHUPPENIES: It's App Engine Test Appendix. DANNY HERMES: OK, appendix, appendix, got it. ROBERT SCHUPPENIES: And the name came from Guido. So I just wanted [? to do that. ?] DANNY HERMES: A benevolent name. ROBERT SCHUPPENIES: Yes. OK, let's look at this. Again, we have an app here. In this app, app.yaml, main.py. And it then has a tests directory. And it includes the aeta library. So if we look at app.yaml, we see whatever comes to tests/ is handled by aeta. And if we look at the tests directory, we just see a bunch of tests there, fail tests, for instance. And now, let's start this app and see what it would look like. So the app, the app itself doesn't tell anything, but it knows where the tests is.
So it points you there. And this is now handled by aeta. And aeta will go through the tests directory and look for all the tests. So this is a list of all the tests it found. This is the package. This is the test module, and then it found those tests classes with those test methods. And then it allows you to run them. And we would just basically say run them all. DANNY HERMES: And are these happening asynchronous via HTTP calls? They're not some sub-process running on App Engine or anything like that? ROBERT SCHUPPENIES: Oh, this one– so what happens this front-end on the website talks to the handler from aeta and says, give me all the tests. And then it responses to all the tests. DANNY HERMES: And then it renders it based on what it receives. ROBERT SCHUPPENIES: Yes. So it's kind of AJAX-y. DANNY HERMES: Yes– ROBERT SCHUPPENIES: And then we saw– DANNY HERMES: –kind of, kind of.
ROBERT SCHUPPENIES: –one test that failed, which is test_fail, which is expected because it even says so. DANNY HERMES: Cool. ROBERT SCHUPPENIES: OK, and all you need to do to use aeta is basically go to, let's see, aeta.googlecode.com, which describes how to use aeta and all that stuff, and put it in your app directory, set up app.yaml, and then set up a test directory. DANNY HERMES: It's not yet on PyPI ready to be pip installed? ROBERT SCHUPPENIES: No. DANNY HERMES: Someday, huh? ROBERT SCHUPPENIES: It will be. And one thing that's also useful in the context of aeta, you don't want to, again, go to a website, click on something, pass the test results. It's manual, expensive, and just like a drag basically. So what we're going to do is we're going to start the application on dev app server, and then– oh, that's small– we can use a local client. And this local client takes just a URL, test– I think it was this one.
And then go to the aeta handler and then query all the– let me just see what's– tests, OK, wrong path. So it's going to work. This should work– goes to the aeta handler and then queries, say, give me all the tests and then we'll run all the tests. And why do we want this in this wrapper? Because we want to test code within an App Engine environment. So we want to test it within either the app server [? environment, ?] so the production environment, or within the SDK. Also– OK, that failed. DANNY HERMES: Maybe you need to be super user or something? ROBERT SCHUPPENIES: Let's see. DANNY HERMES: That's not a hammer for every nail, by the way. I'm just postulating there. AMY UNRUH: Maybe– I'm sorry. ROBERT SCHUPPENIES: It just said local.
DANNY HERMES: There we go. Typo. ROBERT SCHUPPENIES: Typo. And so now we know, OK, it passed. And you can put this together with gaedriver to start your app automatically. And then use aeta, the local client, to run the test. DANNY HERMES: So you're running it from the main directory of this application. ROBERT SCHUPPENIES: Yeah. DANNY HERMES: So where does local client come from? It comes distributed with aeta? ROBERT SCHUPPENIES: Yeah, it comes with aeta. It's a local standalone client. DANNY HERMES: OK, so is there a reason it's at the root of the directory, rather than– ROBERT SCHUPPENIES: Oh, no. Oh, actually it's not the root of the directory. It's just random. We're actually outside of the directory. DANNY HERMES: I see. OK, I got ya. Just so you just put it so it was below the application, so you could actually hit the application or what? ROBERT SCHUPPENIES: So the application, the local client has nothing to do with the application or with a– DANNY HERMES: I realize this.
I'm just curious about why we didn't just go into the aeta directory in the application to run local client? ROBERT SCHUPPENIES: Probably what you– DANNY HERMES: Semantics, folks. Really irrelevant but– ROBERT SCHUPPENIES: What you will do at some point is– and what our test setup also looks like is– we have a bunch of unit tests. We put them in an app wrapper, then deploy those apps with an automated script, based on gaedriver, run the tests with the local client, and then integrate the feedback into our test framework. So it's really totally separate from the app. And one thing we also do is we test it with different environments. So we use it to test it against Python 2.5 environment, Python 2.7, different dev app server setups.
So yeah, this is basically where we use aeta. And those are the three things I wanted to talk about. DANNY HERMES: Super helpful demos, right? If nothing else, just seeing them all together and seeing the quick mocks you can make. It's pretty big. As you could tell with some of my questions, Robert's test_foo is completely superior to my test_foo, even though I consider myself to be something of a Python nut. But these things are actually pretty important if you have a production application, and you're running a business off of it, which hopefully plenty of people watching are. So Amy, what next? What else are we going to do? AMY UNRUH: So thanks so much, Robert. That was great. And if you're like me and put off writing your tests until the end of your app building, maybe this will inspire you, because you can see how easy it is and how great some of this stuff is. So I think we've got a few more questions coming in on the Moderator that are test related.
So I want to apologize in advance to the few questions that are not test related, which I think we might need to get to next time. We'll propagate them along. Let's see. So we've covered the one about webapp2. And you sort of covered this one. But I'll let you answer it further if you want to. For functional tests, is there any recommended setup– BDD, Selenium, any built-in feature in App Engine for this? ROBERT SCHUPPENIES: There's no built-in feature in App Engine, because, like the webtest library we use to do handler testing, there are really great open source projects to do functional testing, so [? drive ?] [? your ?] [? website ?], for example. My personal preference is WebDriver. It has a great API. And it allows you to use different types of browsers as well. And if you use, for example, gaedriver, you can start your app or deploy it and then integrate that with the WebDriver test that hits certain URLs, certain elements on the page and then checks the output.
DANNY HERMES: Sorry. AMY UNRUH: OK, here's another one. And again, you've touched on it a bit. How do you set up your test environments, especially CI with a packaging format that App Engine is delivered in? ROBERT SCHUPPENIES: I'm not sure I quite understand the question. What's CI? AMY UNRUH: So Continuous Integration. ROBERT SCHUPPENIES: So what we do as part of App Engine is also we have applications that are checked in and then we use, for example, gaedriver to, whenever there's a change, run them. For example, with the SDK, run all the tests like WebDriver-based tests and then see whether they still pass. And the same you should be doing probably both for the SDK, but also before you deploy. Because the SDK tries to be as close as possible to the production environment, but it's not quite there.
DANNY HERMES: It's not equivalent. ROBERT SCHUPPENIES: It's not equivalent, but close. So it really makes sense– and we do the same– to test our applications both on the SDK and in the production environment by using tools like gaedriver to deploy them automatically with an API and by using tools like WebDriver to do the functional testing. And since those have APIs, it's easy to write scripts that do that for you, so you can automate it. DANNY HERMES: Also, I guess the second part of this question asked about the packaging. But as we saw with things like gaedriver, you can actually deal with that in your config file. So it's something that as long as you're putting it in the same place, or as long as you know where you put it, you'll to be able to set that in your config.
Next question? Anything else? AMY UNRUH: That is actually the last testing-related question, although I'll briefly address one that's easy to address. Are there plans for a Dart runtime environment? So this, I'm afraid, is our stock answer, that we can't comment on things that aren't on the roadmap. Sorry about that. DANNY HERMES: I can answer more questions [? if you ?] can't comment, if you want. [LAUGHING] AMY UNRUH: Well, it's been sitting there for a while. And there's also a good question about Eclipse, that we hope to– we've been having Python-heavy Hangouts recently. So we hope to have some in the future that focus more on Java. So I think we'll propagate that discussion question to an upcoming Hangout, since we're running a bit short of time. Robert, if people were interested in getting this code, just to play around with it, would it be easy to make it available to people? I mean, would it be easy to give it to me, and I could work on making it available? ROBERT SCHUPPENIES: We can do that.
Also, gaedriver and aeta both have demo apps that come with them. So once you download those libraries, you can just work with them. AMY UNRUH: OK, that's probably sufficient. DANNY HERMES: Are there wikis on these projects? If not, then I would be happy to write it up. ROBERT SCHUPPENIES: Just, again, go to the aeta project page and the gaedriver page. They also have stuff. DANNY HERMES: And they've got wikis. Awesome. Great. AMY UNRUH: OK, great. Thanks so much. DANNY HERMES: Go forth and test. AMY UNRUH: It's really wonderful. All right, see everyone next time. We'll have another one next week, another Hangout. So watch for the announcement on that. DANNY HERMES: Thanks, everyone. ROBERT SCHUPPENIES: Bye. .