It Requires Super Human Discipline to Write Readable Code in Callbacks

Feb 1, 2014 · Comments
Computer ScienceDevelopmentOpen SourceSoapbox

Callbacks can be messy, as Guido Van Rossum said in his recent asyncio talk (@ 56:27 in the YouTube video):

“It requires super human discipline to write readable code in callbacks and if you don’t believe me look at any piece of JavaScript code.” ~ Guido Van Rossum

Fortunately you don’t have to write callbacks if you don’t want to. You need only look to Tornado and Motor

Python has some neat features that let allow generators to be used as coroutines. The Tornado Web Framework includes decorators to make this dead simple. For me, the biggest drawback to using Tornado was the lack of a good asynchronous db driver. Using a blocking db driver sort of defeats the purpose of a non-blocking web server. Fortunately for those of us who don’t love writing callbacks, now there is a good solution.

Motor is a first class asynchronous MongoDB driver written by Jesse Jiryu Davis from MongoDB that will soon take advantage of Futures to present an easy to use, but still asynchronous, interface with MongoDB.

Jesse announced some changes that make using Motor even easier by incorporating Futures into the syntax, just like recent versions of Tornado. We are waiting on MongoDB v2.6 RC, which roadmap indicates will be released on 2/11/14 and the associated version of PyMongo.

In anticipation of this release, I thought it would be fun to look at just how far asynchronous db drivers for Tornado have come. Despite Jesse’s warnings that we should wait until the changes are official, we’ve been using these techniques at MaaSive.net to supercharge developer productivity and increase clarity in our asynchronous code.

In the Beginning

When I started using Motor v0.1 about a year ago this is what trivial DB access looked like for a simple handler that returns some JSON:

If I only ever had to make one db call, and only had to write one callback handler this would not be so bad. The problem is that in reality I need to make several db calls, and may need to wait until two or more are complete to do anything useful. The callbacks get out of hand quickly.

The worst problem is that debugging becomes a nightmare of epic proportions. The entire reason we use Tornado is to enable long-polling with lots of connections at the same time, and it becomes very hard to tell exactly where the problems are when lots of connections are active at the same time. Most developers (myself included) rely on stack-traces to debug our programs, but with callbacks like this you can kiss that nice stack-trace goodbye.

The Present

Motor v0.1.2 has come a long way since that time. Today a trivial DB access looks like this:

Technically you could also write it like this using motor.Op:

This is much, much better than before. Tornado’s tornado.gen module does a great job putting together a “stack-trace” when an exception is thrown and while this is certainly acceptable and useable, it still feels a little clunky. Why do we have to write motor.Op for example?

The motor.Op class exists because motor has a forward looking maintainer who was planning for the Future.

The Future(s)

disclaimer – this api is not released for Motor yet. It will come out with Motor v0.2 along with MongoDB 2.6 RC0 and PyMongo 2.7.

With Futures, we get to write the same thing like this:

The cursor.to_list method returns a Future directly and yielding the Future blocks this coroutine, returning control to the event loop, until the result is ready. It does not get any easier than that.

Developer Cycles are Precious

For many projects the most expensive component is the time put in by the developers, or as I prefer to call them, developer cycles. One of the main reasons we use Python is for the expressiveness of the language. In my experience we can ship more features that our customers want, more quickly by developing in Python than we can in any other language.

Tornado and Motor are making it much easier to deploy asynchronous web servers than it has been in the past. The new AsyncIO library will likely continue to accelerate that trend and that’s a good thing because I really don’t have time to write a bunch of callbacks.

What business needs are you solving with asynchronous web servers?