A post on my Madison Transit API homebrew is long overdue. It was an incredibly fun project and when you see how it has enabled others, you realize how much more compelling it is then the simple SMS app I had originally created.

The API has been out for a long time, but there was very little usage early on. I'm not sure what sparked the flame but in the last few months or so there has been a lot of interest and lot of activity from developers. It dawned on me that I've yet to write about it. And that's a shame because the API is actually the most clever piece of the whole SMSMyBus effort. 

In the beginning

In the beginning, there was a very specific goal of creating an SMS app that delivered real-time arrival estimates. That was it. It was in response to my own need - as a commuter - for such an app. It was also going to be an entry into the Twilio developer contest that was run the week they released the SMS API publicly. 

I knew going in that the Metro didn't provide a formal API or even an XML feed, but I'd done enough HTML screen scraping before to know how to overcome it. Little did I know how tedious that would become with the Metro data. So what started out as a simple texting application turned into a day of cursing the poorly formatted HTML I found on my screen. And every minute of the way I could be heard asking the same question, "Why doesn't the Metro have an API?!?"

When I finished, I swore that I would never let anyone else suffer the pain that I endured so decided to abstract the work I did for the texting app and provide access to my data so other developers could get to the transit data through a standard web service interface. 

Why it's clever

The ugly work of screen scraping was now done and buried in the implementation of the SMS application. But by creating a couple of other web handlers, I could easily make the same scheduling data accessible via a web service call.

For example - http://www.smsmybus.com/api/v1/getarrivals?key=nomar&stopID=1101 - will return real-time arrival estimates for routes traveling through the stop at Main and Carrol on the square. That's the way every developer wants to query data from a web service.

With a little more scraping, I could also find the physical location of every stop (latitude and longitude) so developers could query the location of stops by a stop ID and could also query all nearby stops based on a lat/long coordinate. In the end, there was a pretty impressive looking transit API - simple to use and understand for new developers. And it didn't matter if it was ugly in its technique of screen scraping inside the implementation. It had a clean interface for the developers.

Supported API methods include...

getarrivals - Get real-time arrival estimates for any stop ID with the option of filtering by route ID.

getroutes - Get a list of every route in the Metro system.

getstops - Get the details for every stop a route travels through.

getnearbystops - Get the details for all stops within a given distance of a latitude/longitude.

getstoplocation - Get the location details for a stop given a sop ID.

getvehicles (not implemented) - Get details for a particular vehicle on the road.

getservicebulletins (not implemented) - Get a list of service bulletins in the Metro system.

Why I Built It

As I noted, a big reason for building the API was simply to save the next person the trouble of building up a new set of screen scraping tricks to get the data. But I also knew that most people that have attempted this have stopped at the schedule data. I don't know anything that has gone the extra step of getting the route and stop location data as well. 

More importantly, I wanted to find a way to contribute to the larger mission of opening up a public dataset to make it more accessible. Open data systems are important for lots of reasons, but most importantly, it allows communities to operate more efficiently. And nowhere is that more evident than in transit. Every city, state, and national government in the world should be on a path towards open data right now. 

Unfortunately, Madison has been slow to find its course. My dream was that by creating the API and recruiting some more developers to use it, we'd have enough applications to take back to the city to say, "Look! If you open more data, developers will do great things!"

I also did it for the intellectual challenging of build an API. I've been an API consumer for a long time, but had never constructed one myself. It's an exercise I encourage any API consumer to go through. You'll have a whole new perspective for what it takes to build, document and support an API.

Sample Applications

In theory, all of the original interfaces - SMS, google chat, email, and phone - could have been re-implemented using this API, but I didn't go back and do that. But lots of new apps have been built. For example, the bus kiosk displays hanging in a few local businesses use the API to visualize arrival estimates for nearby stops.

Larry Walker used the API to build a Chumby app and also used it to display arrival estimates on a small LED display on an Arduino. He also built a mobile browser app to more easily access arrival estimates. And I used the API to build a Google gadget so you can get scheduling data in the sidebar of gmail. The attached gallery shows off some of these examples.

Go get started building your own Madison transit app! http://www.smsmybus.com/api/

(download)

Click here to download:

madison-transit-api-homebrew-ddknkkIDEFuFIgmByrqv.zip (317 KB)

Twilio sucked me into one of their contests this weekend. The rules were simple - build an application that leveraged their new Twilio Client, which enables anyone to build phone and SMS interfaces into the browser.

It's a beautiful extension of their existing APIs and really does get you to ask the question, "Why do I need a separate phone device to communicate with others?" 

I didn't have a killer idea or business idea to build on top of this so I decided to do something simple but make it available to anyone to use and extend. I built a browser-based phone that anyone can download and deploy to Google App Engine themselves. All you need is a Twilio account.

The phone has four features. Once you've logged into the application, you can do the following...

• Call anyone using a dial pad right in the browser
• Receive calls in the browser when people call your Twilio number
• Receive calls from friends that login to the app in their browsers
• Record voicemail if you aren't logged in to answer (and send you an SMS to let you know)

Go give me a call - http://phone.gregtracy.com

It was super fun to build and even more fun to use. If you'd like to get your own you can find the project on github - http://github.com/gtracy/phonefree-twilio. It's intended to be a turn-key solution for App Engine. Just deploy and go.

There are lots of opportunities to extend this if you're interested. I'd like to implement voicemail and SMS. I'd like to put a design on this so it looks nice. :) I'd also like to try implementing conferencing so you could have party lines right inside the browser. What else?

(download)

Click here to download:

who-needs-a-phone-the-magic-of-the-twilio-client-BqyBibeqtcHzIgAhabgE.zip (239 KB)

It was about this time a year ago that I built the Madison transit app that delivers real-time arrivals estimates for Metro buses via text messages. Little did I realize that it would turn into such a fun adventure.

I took a look at the logs for the past 52 weeks. Here's SMSMyBus by the numbers...

Total number of users : 138
Total number of requests : 2900
Number of unique stops requested : 318
Most popular stop requested : 1878, Jenifer & Ingersoll

Below is a histogram of the requests each week, and the shape is curious. I've never marketed the service so it's only traveled by word of mouth. The cold weather likely led to the recent jump but I can't explain the general lumpiness throughout.

I also plotted all of the requests on a map using size to indicate the number of requests for each stop. If you're interested, you can explore the map more at http://smsmybus.com/labs/map

Total cost to me? It was a little more than $75 - all of it going towards costs associated with the SMS messaging (Twilio). With the rising popularity, I'd love your donations. Just press the 'donate' button on the site. :)

I built an app over the weekend for the Twilio developer contest called spellitfor.us. The goal of the app was to automate spelling test practices. Allowing kids to practice spelling using the telephone.

On the surface, this project was a combination of scratching a geek itch and laziness. I've been reading words to my kids every week for five years now. And yes it does get boring. But not just for me. The kids are dying for new and more interesting ways to learn. I've tried experimenting with games and I try to have fun using the words in sentences. I'm always looking for news ways to make the chore of homework and studying more interesting and fun. This app was very much in that vein.

While the end product still needs some polish, there are a number of elements that make this a net win. First, they type in the words to create tests (practice in itself). Second, they love to record the words using their own voices and hear them back on the phone. Third, spelling words on the phone makes them feel like they are hip, texting, teenagers.

The features

The app only has a couple of moving parts and the initial implementation was very much focused on an individual practicing.

• Create a spelling test - Let anyone bundle a set of words into a "test", apply a grade level, and name it. I wound up adding a record option for each word as well. This gives the kids a chance to record their own voice.
• Quiz yourself on those tests - Let anyone take a test by responding to prompts on the phone.

There are a lot of community/social opportunities to get kids more engaged. Challenges, leader boards, average scoring, analytics on tricky words, etc. All for another day...

The Twilio magic

The initial goal was to process 100% of the test via voice. I thought I could use my pseudo-synchronous transcription hack and have the kids spell the words one letter at a time - "B-A-N-A-N-A". But I quickly discovered that the transcription service doesn't do letters. It likes words. So I turned to Plan B which uses the <Gather> verb to have the kids use the phone keypad to spell out the word.

The two new things for me on this adventure were the call flow dynamics as well as the call-out feature from the browser. While testing, I never tired from pressing my little phone button in the browser UI and having my phone ring. It brought a smile to my face every time. :)

If you're interested in seeing the nitty gritty details of how I built it, I've posted the code - a Google App Engine project - to github. Not well commented with plenty of shortcuts, but as I noted, I built that in a day. I'm putting it out there so there are more examples for others to learn from. The Twilio API code itself should be a pretty example for you, however.

Now go test your spelling - http://spellitfor.us!

(download)

Click here to download:

Hacking_education_with_Twilio_.zip (60 KB)

I've been exploring more of the Twilio API recently. Since creating Ringerous and SMSMyBus, I've begun looking into the transcription services. But I immediately ran into a challenge. The transcription service was never designed to be synchronous. So although the recording of a call is immediately available, the transcription appears to be intended for offline processing.

You can get a lot of mileage out of this, but there are applications where it would be nice to have access to the transcription immediately. Especially when accuracy is important. I was able to use the following pattern to achieve a true synchronous voice transcription...

1. Inbound call handler

The main handler for the initial inbound phone calls produces simple TwiML that prompts the caller and records what they say. Note that transcription is enabled and I specify a separate callback (which is asynchronous).

2. Recording handler

The recording handler processes the callback when the recording has completed. This is synchronous and thus occurs while the user is still on the original call. You can do whatever you'd like with the recording, but note that the transcription is not ready yet.

The most important thing to do during this step is to record the active CallSid so you can reference it in step three.

You can do whatever you'd like with the caller at this point, but the bottom line is they need to wait for the transcription to complete. So one thing to do is simply play them some pretty music. :)

3. Respond to the transcription

When Twilio's transcription engine completes, you'll get the callback you specified in the record verb in step one. Take a moment to store away the transcription text.

Now use the stored CallSid from step two along with the REST interface to interrupt the caller that is listening to that pretty music you're serving up for them. Specify yet another handler URL when you interrupt the call. You can serve up brand new TwiML that way.

4. Interrupt handler

The interrupt via the REST interface gives you an opportunity to treat the caller to any workflow you'd like. This is a great chance to ask them if the transcription is correct and gather their input - either via voice or textpad - and react accordingly. In this case, I use the Say verb to read back the transcription I've stored in step three.

That's it. Synchronous voice transcription. Now all you need to do is add your favorite API to the verificationHandler so you can hook it up to another app like Twitter, Posterous, Google Search, etc.

Do you know of an easier way to do this? If so, please share...

Over the last few days, I've been camped out at a local pool for Madison's annual All City Swimming Championships. Twelve teams and 1,703 swimmers. A fun but exhausting couple of days. Managing your kids is a two-part challenge. The first part is actually getting them to the block on time for their race. The second part is trying to sort through the 100+ swimmers in the event to determine how well they did.

Tracy, Anna finished event 21 in 40.26, ranked 142

I had a huge smile on my face this morning when I scrolled through my Twitter timeline and found @gl33p's pictures of the new display at Mother Fools Coffeehouse. Coffeehouse patrons are now enjoying real time arrival estimates for the Madison Metro buses passing by at the corner of Ingersoll and Jennifer courtesy of SMSMyBus!

Two weeks ago when Twilio announced their developer contest for their new SMS API, I decided to build a mobile application that let me query the Madison Metro bus system to determine when my bus would arrive.

Although the entry did not win the contest, it was named mashup of the day last week at Programmable Web! It's called SMS My Bus, and if you live in Madison and ride the bus I encourage you to take advantage of it! You can find details about it here...

http://www.smsmybus.com

The basic architecture of the application is straight forward. SMS messages are sent to my Twilio phone number and Twilio routes them to my server via HTTP POST requests. I do a schedule look-up based on the user's input and return the results.

The tricky parts stem primarily from the fact that:

1. The Madison Metro doesn't actually provide web services for this data. The consequences for an app like mine is that I need to do a bit of screen scraping to find the data I'm after.

2. I chose to deploy this app using Google App Engine, and URL scraping can become a show stopper since GAE is resource limiting for every request that runs. GAE will only let you a single request for about 30 seconds.

Needless to say, I would love it if my fine city of Madison would join the Gov 2.0 movement, and open up more of its rich data via standard web services

In the meantime... I needed a solution that would allow me to gather disparate data across many, many URLS. As an example, the busiest stop in the Metro system has 34 buses passing through it. I may need to grab 34 different web pages to begin to piece together the schedule as it relates to the caller at that stop.

I took advantage of GAE's Task Queue API and memcache counters to tackle this problem by farming out autonomous jobs that find the next available bus per route per stop, and at the end aggregating the results.

Admittedly, this is not advanced Computer Science. But I hope other App Engine developers can find some use in the pattern.

1. Define my task queues

I used two different task queues to manage the process. One, called aggregation, that queried individual routes at a stop. And another, called aggregationSMS, that pieced the results together for the return SMS message.

 
- name: aggregation 
  rate: 20/s 
  bucket_size: 1 
 
- name: aggregationSMS 
  rate: 10/s 
  bucket_size: 1 

2. Spawn tasks

When an SMS request arrives, the request handler parses the input to determine the request parameters. If the request does not include a specific bus route, I'll query my route table for every route that passes through the respective stop. This table contains URLs for the the real time arrival estimates.

I loop over the result set and create new tasks for the aggregation task queue.

 
    q = db.GqlQuery("SELECT * FROM RouteListing WHERE stopID = :1",stopID) 
    routeQuery = q.fetch(100) 
    if len(routeQuery) > 0: 
        # create a counter for a universally unique caller ID 
        memcache.add(sid, 0) 
 
        # loop over every route at this stop 
        for r in routeQuery: 
          # the unique counter for this caller's request 
          counter = memcache.incr(sid) 
 
          # spawn a task for this stop/route tuple 
          task = Task(url='/aggregationtask', 
                      params={'sid':sid, 
                              'stop':stopID, 
                              'route':r.route, 
                              'direction':r.direction, 
                              'url':r.scheduleURL, 
                              'caller':caller 
                              }) 
          task.add('aggregation') 
    else: 
        # do some error handling 

3. Define the task handlers

There are two task handlers. One to tackle the smallest job of determining the schedule for an individual bus at a stop. And one to piece all of these results together when the system is ready to reply to the caller.

The task handler, AggregationHandler, does the specific work to scrape the scheduling information from the bus system's site. The handler does three things.

• Scrape the web page to find the next stop time.
• Store the results in the datastore
• Decrement the memcache counter for this transaction

Many of the implementation details have been stripped out of the following code snippet...

 
 
class AggregationHandler(webapp.RequestHandler): 
 
 def post(self): 
 # extract the parameters for this task 
 sid = self.request.get('sid') 
 directionID = self.request.get('direction') 
 # more inputs as well... 
 
 # 1. fetch the real time data 
 result = urlfetch.fetch(scheduleURL) 
 
 # scrape the page 
 textBody = result.getNextTime() 
 
 # 2. store these results in the datastore 
 stop = BusStopAggregation() 
 stop.stopID = stopID 
 stop.routeID = routeID 
 stop.sid = sid # the sid identifies the caller's transaction 
 stop.text = textBody 
 stop.put() 
 
 # 3. decrement the counter 
 counter = memcache.decr(sid) 
 
 # if we've completed the scraping, create a task to 
 # piece the results together. 
 if counter == 0: 
   task = Task(url='/aggregationSMStask', 
                     params={'sid':sid,'caller':caller}) 
   task.add('aggregationSMS') 
 
 # delete the counter for this transaction 
 memcache.delete(sid) 
 
 return 
 

The task handler, AggregationSMSHandler, does the job of piecing the results together. It relies on the unique SID for a caller's transaction to query the datastore and find the scheduling details.

 
 
class AggregationSMSHandler(webapp.RequestHandler): 
 
 def post(self): 
 # extract the task's inputs 
 sid = self.request.get('sid') 
 phone = self.request.get('caller') 
 
 # sort the results by time to find soonest upcoming stops 
 q = db.GqlQuery("SELECT * FROM BusStopAggregation WHERE sid = :1 ORDER BY time", sid) 
 
 # we'll only send the next four stops in the reply message 
 routeQuery = q.fetch(4) 
 stopID = routeQuery[0].stopID 
 textBody = "Stop: %s\n" % routeQuery[0].stopID 
 for r in routeQuery: 
 textBody += "Route %s: " % r.routeID + " %s" % r.text + "\n" 
 else: 
 textBody = "Doesn't look good... Your bus isn't running right now!" 
 
 # send off the result via the twilio API 
 outboundSMS(phone, textBody) 
 

Results

This pattern allowed me to almost completely mitigate the DeadlineExceededExceptions on GAE. I've yet to see a timeout problem inside the app. It's always possible that a single task could take too long, but if a task fails, it will re-queue itself and run again until it succeeds.

It's worth pointing out that another use of the Task Queue that I used but didn't show in the code snippets were for other repeated, remote tasks. For example, when I interface with the Twilio API, I do that by spawning a task to do the work. Likewise, I log Twilio events in the datastore on their own task queue as well.

To celebrate being named one of the Best New Mashups over at Programmable Web this week, I'm going to list the top five use cases for Ringerous.

Family blogging

Now my kids are calling in to the blog from their sporting events and from the backyard to announcement their latest and greatest personal achievements. All in their own voices.

Mobile blogging (for the rest of us)

Podcasting in the classroom

Combine this with the drop-dead simple podcasting you can do with iTunes, and you get a great distribution model for the students and teachers as well.

Bring emotion to your posts

Public voicemail

How are you using Ringerous?

http://www.ringerous.com