Greg Tracy

The Google App Engine developer community is a hot mess this week over the new pricing plan for the platform. And for good reason. Many developers are seeing their hosting expenses going up by as much as 500%.

If you're looking for a post that is trashing the App Engine team, you can move along. You won't find it here. These guys are smart and considerate. If you spend any time interacting with them on StackOverflow, email or in person at Google IO, you understand this. In fact, just using the platform for a project you can appreciate their outlook and passion for their product and users. That's not to say they don't have room to improve. But enough with the negativity already!

Effects of new pricing on my projects

There have been a lot of people posting about their apps and revealing the effects of the new pricing on them. I wanted to do the same as a reference point. Note that my use of App Engine has primarily been for personal projects. Some have web front-ends, some have SMS interfaces, some are just based on background tasks and others come and go while I experiment with ideas or calendar events. I still think most of these experiments are well suited for App Engine, but I need to take a hard look at the more successful apps to figure out a long-term strategy because they are not scaling well with the new pricing plan.

I'll share two examples - both philanthropic projects - comparing the effects of the new pricing.

Astronomy Picture of the Day

This app had originally been written in Perl as a grad student and was hosted at the University of Wisconsin. I decided to port the application as the vehicle for learning App Engine and Python so it was the first app I ever wrote on the platform. It's primarily a background app. Every afternoon it runs a job that scrapes the contents of the APOD site and packages it into an email and sends it off to all subscribers. There's a simple web frontend that lets anyone sign up. There are currently 1900+ subscribers.

The app is free to run on the platform today and will cost $0.19/day - or $0.03 per user per year - after the price changes. 100% of those costs can be attributed to the use of the Mail API.

My only complaint about this app is that the change seems extreme. Going from 2,000 free emails to 100 feels like an attempt to curb the spamming community. And for the charity projects like this, all of the good net citizens are the losers with this change.

SMSMyBus

This app was originally built to provide a better interface for the Madison Metro bus service. It provides real-time arrival times for buses via SMS, chat, email and phone. But then it blossomed into a full-featured API for the Metro for other developers.

The app costs $0.01/day to operate today (excluding the SMS interface). It is estimated to cost $6.79/day after the price change. $2,478/year. Yah. That's a whopping 67,800% increase. Shebang.

The root cause of essentially all the cost can be attributed to the main API call that returns arrival times at a particular bus stop - getarrivals - and some of the clients call this repeatedly (like every two minutes). It is also where the confusion starts for me with respect to the new pricing.

Frontend instance hours

Frontend instance hours is projected to be $5.68/day, 84% of the bill. This represents the platform's transition from billing for CPU usage to billing for the contention of instance usage. I get it that they need to do this. They were using the wrong resource metric for monitoring before. 

But how do I go from a $0.00 cost for resource consumption to $5.68/day?!? That kind of increment just feels insane. How about $0 to $0.50? Or $0 to $1?

Datastore writes

Datastore writes is projected to be $1.00/day, 14% of the bill. This is harder for me to resolve for a couple of reasons. First, I can't find any cost under the current pricing plan for these operations even though the app's profile is fairly consistent. So I struggle, conceptually, with how this has suddenly become an issue for the app.

Second, $1/day equates to 1M writes/day in the datastore and I simply can't figure out where all of those writes are coming from. My back of the napkin math shows 40,000 writes. I'm totally baffled by this projection. 

The rest of it

The rest of the projected cost is a combination of storage and datastore read operations. I can eliminate the former if I simply store less data I wanted to use for analytics. It saves me money, but in the end, ignoring some of the data hurts the developers that use the API.

Optimizing

Now it's my job to go in and take another stab at optimizing the code and start with the getarrivals API call. I thought I had good habits with this so I was a little embarrassed when I found an obvious hole in the query path for route listings. There's a fairly repetitive query that was not being memcached - oops! Now fixed.

The second thing I'm experimenting with is the application's instance configuration. By default, I was letting the platform's scheduler determine my load patterns and create new instances whenever necessary. But I've made two changes. First, I took the scheduler out of 'auto' mode and set the maximum number of idle instances to one, and I've cranked up the minimum latency for the pending request queue to 250ms. In theory, each of these changes should drive the cost down because I should be using less frontend instance time throughout the day.

Let's see what happens! As I do my part with optimization, I'd like to see the App Engine do their part and move to the middle as well. :)

What to do next

I'm guessing that the App Engine platform simply priced things wrong the first time. I think the concept of platform as a service that exploits existing Google infrastructure was a smart, but geeky idea that was poorly modeled or had bad assumptions about its use/abuse. Ironically, the idea didn't scale well and they've been forced to admit that early assumptions on how to price it were just wrong.

The good part about this move... developers are forced to take a deep dive into optimization. Something i've written about before and have been doing again since the clock started on the pricing changes. This not only makes for a better platform for Google and project sharing the resources, but it makes for a better net as a whole. Faster is better.

The bad part... 

• Developers will be forced to dead pool worthy projects that don't have a business model. 
• Developers will be forced to port apps to other platforms. That could be a painful pill to swallow for developers when they aren't money making projects
• Developers may be sacrificing analytics to avoid datastore bloat and access charges.

What the App Engine team should do about it

• Provide better pricing structures for philanthropic and open source projects. App Engine is a great platform for these things and it provides a great playground for developers to support important projects at a low cost while also learning about a platform they can adopt for larger, commercial project down the road. They've hinted at this but will they do it? - http://code.google.com/appengine/kb/postpreviewpricing.html#special_programs_...
• Provide more runway for optimization. A couple of weeks to get the sleeves rolled up and optimize their apps just isn't enough time.
• Provide better analysis tools to highlight problems
• Take baby steps. Must they really take these giant leaps in pricing?
• Roll out Python 2.7 to support concurrent requests in Python projects.

In the process of writing this post I found some great resources...

• A nice FAQ from App Engine on the new pricing plan - http://code.google.com/appengine/kb/postpreviewpricing.html
• A nice resource to help you understand how to manage your app's instance utilization; the real cost bottleneck - http://code.google.com/appengine/docs/adminconsole/performancesettings.html
• An excellent post from another developer tackling this problem - http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/

A post on my Madison Transit API homebrew is long overdue. It was an incredibly fun project and when you see how it has enabled others, you realize how much more compelling it is then the simple SMS app I had originally created.

The API has been out for a long time, but there was very little usage early on. I'm not sure what sparked the flame but in the last few months or so there has been a lot of interest and lot of activity from developers. It dawned on me that I've yet to write about it. And that's a shame because the API is actually the most clever piece of the whole SMSMyBus effort. 

In the beginning

In the beginning, there was a very specific goal of creating an SMS app that delivered real-time arrival estimates. That was it. It was in response to my own need - as a commuter - for such an app. It was also going to be an entry into the Twilio developer contest that was run the week they released the SMS API publicly. 

I knew going in that the Metro didn't provide a formal API or even an XML feed, but I'd done enough HTML screen scraping before to know how to overcome it. Little did I know how tedious that would become with the Metro data. So what started out as a simple texting application turned into a day of cursing the poorly formatted HTML I found on my screen. And every minute of the way I could be heard asking the same question, "Why doesn't the Metro have an API?!?"

When I finished, I swore that I would never let anyone else suffer the pain that I endured so decided to abstract the work I did for the texting app and provide access to my data so other developers could get to the transit data through a standard web service interface. 

Why it's clever

The ugly work of screen scraping was now done and buried in the implementation of the SMS application. But by creating a couple of other web handlers, I could easily make the same scheduling data accessible via a web service call.

For example - http://www.smsmybus.com/api/v1/getarrivals?key=nomar&stopID=1101 - will return real-time arrival estimates for routes traveling through the stop at Main and Carrol on the square. That's the way every developer wants to query data from a web service.

With a little more scraping, I could also find the physical location of every stop (latitude and longitude) so developers could query the location of stops by a stop ID and could also query all nearby stops based on a lat/long coordinate. In the end, there was a pretty impressive looking transit API - simple to use and understand for new developers. And it didn't matter if it was ugly in its technique of screen scraping inside the implementation. It had a clean interface for the developers.

Supported API methods include...

getarrivals - Get real-time arrival estimates for any stop ID with the option of filtering by route ID.

getroutes - Get a list of every route in the Metro system.

getstops - Get the details for every stop a route travels through.

getnearbystops - Get the details for all stops within a given distance of a latitude/longitude.

getstoplocation - Get the location details for a stop given a sop ID.

getvehicles (not implemented) - Get details for a particular vehicle on the road.

getservicebulletins (not implemented) - Get a list of service bulletins in the Metro system.

Why I Built It

As I noted, a big reason for building the API was simply to save the next person the trouble of building up a new set of screen scraping tricks to get the data. But I also knew that most people that have attempted this have stopped at the schedule data. I don't know anything that has gone the extra step of getting the route and stop location data as well. 

More importantly, I wanted to find a way to contribute to the larger mission of opening up a public dataset to make it more accessible. Open data systems are important for lots of reasons, but most importantly, it allows communities to operate more efficiently. And nowhere is that more evident than in transit. Every city, state, and national government in the world should be on a path towards open data right now. 

Unfortunately, Madison has been slow to find its course. My dream was that by creating the API and recruiting some more developers to use it, we'd have enough applications to take back to the city to say, "Look! If you open more data, developers will do great things!"

I also did it for the intellectual challenging of build an API. I've been an API consumer for a long time, but had never constructed one myself. It's an exercise I encourage any API consumer to go through. You'll have a whole new perspective for what it takes to build, document and support an API.

Sample Applications

In theory, all of the original interfaces - SMS, google chat, email, and phone - could have been re-implemented using this API, but I didn't go back and do that. But lots of new apps have been built. For example, the bus kiosk displays hanging in a few local businesses use the API to visualize arrival estimates for nearby stops.

Larry Walker used the API to build a Chumby app and also used it to display arrival estimates on a small LED display on an Arduino. He also built a mobile browser app to more easily access arrival estimates. And I used the API to build a Google gadget so you can get scheduling data in the sidebar of gmail. The attached gallery shows off some of these examples.

Go get started building your own Madison transit app! http://www.smsmybus.com/api/

(download)

Click here to download:

madison-transit-api-homebrew-ddknkkIDEFuFIgmByrqv.zip (317 KB)

Twilio sucked me into one of their contests this weekend. The rules were simple - build an application that leveraged their new Twilio Client, which enables anyone to build phone and SMS interfaces into the browser.

It's a beautiful extension of their existing APIs and really does get you to ask the question, "Why do I need a separate phone device to communicate with others?" 

I didn't have a killer idea or business idea to build on top of this so I decided to do something simple but make it available to anyone to use and extend. I built a browser-based phone that anyone can download and deploy to Google App Engine themselves. All you need is a Twilio account.

The phone has four features. Once you've logged into the application, you can do the following...

• Call anyone using a dial pad right in the browser
• Receive calls in the browser when people call your Twilio number
• Receive calls from friends that login to the app in their browsers
• Record voicemail if you aren't logged in to answer (and send you an SMS to let you know)

Go give me a call - http://phone.gregtracy.com

It was super fun to build and even more fun to use. If you'd like to get your own you can find the project on github - http://github.com/gtracy/phonefree-twilio. It's intended to be a turn-key solution for App Engine. Just deploy and go.

There are lots of opportunities to extend this if you're interested. I'd like to implement voicemail and SMS. I'd like to put a design on this so it looks nice. :) I'd also like to try implementing conferencing so you could have party lines right inside the browser. What else?

(download)

Click here to download:

who-needs-a-phone-the-magic-of-the-twilio-client-BqyBibeqtcHzIgAhabgE.zip (239 KB)

I built an app over the weekend for the Twilio developer contest called spellitfor.us. The goal of the app was to automate spelling test practices. Allowing kids to practice spelling using the telephone.

On the surface, this project was a combination of scratching a geek itch and laziness. I've been reading words to my kids every week for five years now. And yes it does get boring. But not just for me. The kids are dying for new and more interesting ways to learn. I've tried experimenting with games and I try to have fun using the words in sentences. I'm always looking for news ways to make the chore of homework and studying more interesting and fun. This app was very much in that vein.

While the end product still needs some polish, there are a number of elements that make this a net win. First, they type in the words to create tests (practice in itself). Second, they love to record the words using their own voices and hear them back on the phone. Third, spelling words on the phone makes them feel like they are hip, texting, teenagers.

The features

The app only has a couple of moving parts and the initial implementation was very much focused on an individual practicing.

• Create a spelling test - Let anyone bundle a set of words into a "test", apply a grade level, and name it. I wound up adding a record option for each word as well. This gives the kids a chance to record their own voice.
• Quiz yourself on those tests - Let anyone take a test by responding to prompts on the phone.

There are a lot of community/social opportunities to get kids more engaged. Challenges, leader boards, average scoring, analytics on tricky words, etc. All for another day...

The Twilio magic

The initial goal was to process 100% of the test via voice. I thought I could use my pseudo-synchronous transcription hack and have the kids spell the words one letter at a time - "B-A-N-A-N-A". But I quickly discovered that the transcription service doesn't do letters. It likes words. So I turned to Plan B which uses the <Gather> verb to have the kids use the phone keypad to spell out the word.

The two new things for me on this adventure were the call flow dynamics as well as the call-out feature from the browser. While testing, I never tired from pressing my little phone button in the browser UI and having my phone ring. It brought a smile to my face every time. :)

If you're interested in seeing the nitty gritty details of how I built it, I've posted the code - a Google App Engine project - to github. Not well commented with plenty of shortcuts, but as I noted, I built that in a day. I'm putting it out there so there are more examples for others to learn from. The Twilio API code itself should be a pretty example for you, however.

Now go test your spelling - http://spellitfor.us!

(download)

Click here to download:

Hacking_education_with_Twilio_.zip (60 KB)

Over the last few days, I've been camped out at a local pool for Madison's annual All City Swimming Championships. Twelve teams and 1,703 swimmers. A fun but exhausting couple of days. Managing your kids is a two-part challenge. The first part is actually getting them to the block on time for their race. The second part is trying to sort through the 100+ swimmers in the event to determine how well they did.

Tracy, Anna finished event 21 in 40.26, ranked 142

I had a huge smile on my face this morning when I scrolled through my Twitter timeline and found @gl33p's pictures of the new display at Mother Fools Coffeehouse. Coffeehouse patrons are now enjoying real time arrival estimates for the Madison Metro buses passing by at the corner of Ingersoll and Jennifer courtesy of SMSMyBus!

Click here to download:

web608_app_engine_overview.pdf (791 KB)

(download)

Click here to download:

web608_app_engine_overview.pdf (791 KB)

I gave a tech talk tonight on my experience with Google App Engine for the web608 group. I hope these talks continue - it's great for this city to have more events like these.

Link to Google presentation... http://docs.google.com/present/view?id=dhffp9s2_24xt7nmtgz&interval=5

Two weeks ago when Twilio announced their developer contest for their new SMS API, I decided to build a mobile application that let me query the Madison Metro bus system to determine when my bus would arrive.

Although the entry did not win the contest, it was named mashup of the day last week at Programmable Web! It's called SMS My Bus, and if you live in Madison and ride the bus I encourage you to take advantage of it! You can find details about it here...

http://www.smsmybus.com

The basic architecture of the application is straight forward. SMS messages are sent to my Twilio phone number and Twilio routes them to my server via HTTP POST requests. I do a schedule look-up based on the user's input and return the results.

The tricky parts stem primarily from the fact that:

1. The Madison Metro doesn't actually provide web services for this data. The consequences for an app like mine is that I need to do a bit of screen scraping to find the data I'm after.

2. I chose to deploy this app using Google App Engine, and URL scraping can become a show stopper since GAE is resource limiting for every request that runs. GAE will only let you a single request for about 30 seconds.

Needless to say, I would love it if my fine city of Madison would join the Gov 2.0 movement, and open up more of its rich data via standard web services

In the meantime... I needed a solution that would allow me to gather disparate data across many, many URLS. As an example, the busiest stop in the Metro system has 34 buses passing through it. I may need to grab 34 different web pages to begin to piece together the schedule as it relates to the caller at that stop.

I took advantage of GAE's Task Queue API and memcache counters to tackle this problem by farming out autonomous jobs that find the next available bus per route per stop, and at the end aggregating the results.

Admittedly, this is not advanced Computer Science. But I hope other App Engine developers can find some use in the pattern.

1. Define my task queues

I used two different task queues to manage the process. One, called aggregation, that queried individual routes at a stop. And another, called aggregationSMS, that pieced the results together for the return SMS message.

 
- name: aggregation 
  rate: 20/s 
  bucket_size: 1 
 
- name: aggregationSMS 
  rate: 10/s 
  bucket_size: 1 

2. Spawn tasks

When an SMS request arrives, the request handler parses the input to determine the request parameters. If the request does not include a specific bus route, I'll query my route table for every route that passes through the respective stop. This table contains URLs for the the real time arrival estimates.

I loop over the result set and create new tasks for the aggregation task queue.

 
    q = db.GqlQuery("SELECT * FROM RouteListing WHERE stopID = :1",stopID) 
    routeQuery = q.fetch(100) 
    if len(routeQuery) > 0: 
        # create a counter for a universally unique caller ID 
        memcache.add(sid, 0) 
 
        # loop over every route at this stop 
        for r in routeQuery: 
          # the unique counter for this caller's request 
          counter = memcache.incr(sid) 
 
          # spawn a task for this stop/route tuple 
          task = Task(url='/aggregationtask', 
                      params={'sid':sid, 
                              'stop':stopID, 
                              'route':r.route, 
                              'direction':r.direction, 
                              'url':r.scheduleURL, 
                              'caller':caller 
                              }) 
          task.add('aggregation') 
    else: 
        # do some error handling 

3. Define the task handlers

There are two task handlers. One to tackle the smallest job of determining the schedule for an individual bus at a stop. And one to piece all of these results together when the system is ready to reply to the caller.

The task handler, AggregationHandler, does the specific work to scrape the scheduling information from the bus system's site. The handler does three things.

• Scrape the web page to find the next stop time.
• Store the results in the datastore
• Decrement the memcache counter for this transaction

Many of the implementation details have been stripped out of the following code snippet...

 
 
class AggregationHandler(webapp.RequestHandler): 
 
 def post(self): 
 # extract the parameters for this task 
 sid = self.request.get('sid') 
 directionID = self.request.get('direction') 
 # more inputs as well... 
 
 # 1. fetch the real time data 
 result = urlfetch.fetch(scheduleURL) 
 
 # scrape the page 
 textBody = result.getNextTime() 
 
 # 2. store these results in the datastore 
 stop = BusStopAggregation() 
 stop.stopID = stopID 
 stop.routeID = routeID 
 stop.sid = sid # the sid identifies the caller's transaction 
 stop.text = textBody 
 stop.put() 
 
 # 3. decrement the counter 
 counter = memcache.decr(sid) 
 
 # if we've completed the scraping, create a task to 
 # piece the results together. 
 if counter == 0: 
   task = Task(url='/aggregationSMStask', 
                     params={'sid':sid,'caller':caller}) 
   task.add('aggregationSMS') 
 
 # delete the counter for this transaction 
 memcache.delete(sid) 
 
 return 
 

The task handler, AggregationSMSHandler, does the job of piecing the results together. It relies on the unique SID for a caller's transaction to query the datastore and find the scheduling details.

 
 
class AggregationSMSHandler(webapp.RequestHandler): 
 
 def post(self): 
 # extract the task's inputs 
 sid = self.request.get('sid') 
 phone = self.request.get('caller') 
 
 # sort the results by time to find soonest upcoming stops 
 q = db.GqlQuery("SELECT * FROM BusStopAggregation WHERE sid = :1 ORDER BY time", sid) 
 
 # we'll only send the next four stops in the reply message 
 routeQuery = q.fetch(4) 
 stopID = routeQuery[0].stopID 
 textBody = "Stop: %s\n" % routeQuery[0].stopID 
 for r in routeQuery: 
 textBody += "Route %s: " % r.routeID + " %s" % r.text + "\n" 
 else: 
 textBody = "Doesn't look good... Your bus isn't running right now!" 
 
 # send off the result via the twilio API 
 outboundSMS(phone, textBody) 
 

Results

This pattern allowed me to almost completely mitigate the DeadlineExceededExceptions on GAE. I've yet to see a timeout problem inside the app. It's always possible that a single task could take too long, but if a task fails, it will re-queue itself and run again until it succeeds.

It's worth pointing out that another use of the Task Queue that I used but didn't show in the code snippets were for other repeated, remote tasks. For example, when I interface with the Twilio API, I do that by spawning a task to do the work. Likewise, I log Twilio events in the datastore on their own task queue as well.