Rapid Web Development with Tornado Web and MongoDB

Ikai LanTwitter: @ikai

Friday, June 10, 2011

About the speaker

• Developer Relations at Google

• Software Engineering background

• Lives in San Francisco, CA

• Writes Java, Python on a day to day basis

• Twitter: @ikai

Friday, June 10, 2011

This talk

• Why Tornado Web and MongoDB?

• Whirlwind tour of Tornado (get it?!)

• Intro to MongoDB

• Brief look at a demo app

Friday, June 10, 2011

What I won’t be talking about

• Scalability - loaded topic

• Reliability - another loaded topic

Friday, June 10, 2011

So why tornado/mongo?

Friday, June 10, 2011

Joy.Friday, June 10, 2011

Where does this joy come from?

Friday, June 10, 2011

Freedom.Friday, June 10, 2011

So really, my talk should have been called “The

Joy of Tornado Web and Mongo DB”

Friday, June 10, 2011

Intangibles

• Tornado - fast development, very few rules

• MongoDB - freedom from a predefined schema

• Python, of course. Dynamically typed variables, duck typing plus everything else that is cool about the language

Friday, June 10, 2011

Tornado Web history

• Open Sourced by Facebook

• ... formerly of FriendFeed

• ... former Googlers (that’s why the framework looks like App Engine’s webapp - they wrote it)

• Tornado Web powers FriendFeed

Friday, June 10, 2011

A whirlwind tourFriday, June 10, 2011

Tornado features

• Out of the box auth with Twitter, Facebook, Google and FriendFeed

• Out of box support for localized strings

• Simple templating that allows arbitrary Python code

• Asynchronous requests

• Small codebase

Friday, June 10, 2011

Simple handlersimport tornado.ioloopimport tornado.web

class HelloHandler(tornado.web.RequestHandler): def get(self): self.write("Hello, world :)")

class GoodbyeHandler(tornado.web.RequestHandler): def get(self): self.render("goodbye.html", message=”Goodbye, world”)

application = tornado.web.Application([ (r"/hello", HelloHandler), (r"/goodbye", GoodbyeHandler), ])

if __name__ == "__main__": application.listen(8888) tornado.ioloop.IOLoop.instance().start()

Friday, June 10, 2011

Asynchronous natureclass MainHandler(tornado.web.RequestHandler):

@tornado.web.asynchronous def get(self): http = tornado.httpclient.AsyncHTTPClient() http.fetch("http://friendfeed-api.com/v2/feed/bret", callback=self.on_response)

def on_response(self, response): if response.error: raise tornado.web.HTTPError(500) json = tornado.escape.json_decode(response.body) self.write("Fetched " + str(len(json["entries"])) + " entries " "from the FriendFeed API") self.finish()

Friday, June 10, 2011

Unrestrictive templating {% for student in [p for p in people if p.student and p.age > 23] %} <li>{{ escape(student.name) }}</li> {% end %}

# Sample of arbitrary functions in templates def add(x, y): return x + y template.execute(add=add)

### The template {{ add(1, 2) }}

Friday, June 10, 2011

Lies, damn lies and benchmarks

Friday, June 10, 2011

Saving dataFriday, June 10, 2011

MongoDB is an object database

• No schema - anything that can be a JSON object can be stored

• You can define new collections on the fly

Friday, June 10, 2011

Getting started with Mongo in 4 steps

• Download a binary for your system

• Create a data directory (mkdir -p /data/db)

• Run mongod

• Install pymongo

• ... start writing code!

Friday, June 10, 2011

Pymongo code sampleconnection = pymongo.Connection()database = connection["your_database"]

def get_or_create_location_by_id(location_id): """ Attempts to fetch location data from database. If it doesn't exist, create it. Note that this is NOT concurrency safe. """ location_data = database["locations"].find_one({ "_id" : location_id }) if location_data is None: location_data = { "_id" : location_id, "guards": [], "owner" : None, "history" : [], "last_extort_time" : None } database.location.save(location_data, safe=True) return location_data

Friday, June 10, 2011

Pymongo queries# Add Tyson as a guard to every location owned by “Joe Smith”locations = database["locations"].find({ "owner" : "Joe Smith" })for location in locations: location["guards"].append("Tyson") database["locations"].save(location)

# Find everyone who has Tyson as a guardlocations = database["locations"].find({"guards" : "Tyson"})

# Find everyone who has *ONLY* Tyson as a guardlocations = database["locations"].find({"guards" : ["Tyson"]}) # Note []s

# Find everyone who has Tyson as a guard whose owner is “Ikai Lan”locations = database["locations"].find({"guards" : "Tyson", "owner" : "Ikai Lan" })

Friday, June 10, 2011

Mold as you go alongFriday, June 10, 2011

Freedom from

• ... waiting to get started

• ... predefining a schema

• ... worrying about complex data structures that don’t fit well into the SQL box. Anything that is JSON can be stored!

Friday, June 10, 2011

More MongoDB features

• Simple Geospatial indexing

• GridFS - distributed filesystem running on MongoDB

• Out of the box mapreduce

• Out of the box replication, data partitioning

Friday, June 10, 2011

Putting it all together

• MobSquare, a location based game that uses the Facebook API.

• Mashup: Mafia Wars + FourSquare

• https://github.com/ikai/mobsquare-demo

• “Check in” to locations, take them over, extort money and fight other gangs

Friday, June 10, 2011

Why are Tornado/Mongo a fit?

• If I haven’t said it enough yet, they’re fun

• Facebook API performance varies - asynchronous so we can serve requests while waiting

• Highly structured player data

Friday, June 10, 2011

OAuth 2 upgrade flowclass OnLoginHandler(tornado.web.RequestHandler):

@tornado.web.asynchronous def get(self): # Store this somewhere code = self.get_argument("code") access_token_url = ACCESS_TOKEN_URL_TPL + code client = httpclient.AsyncHTTPClient() client.fetch(access_token_url, self.on_fetched_token) def on_fetched_token(self, response): """ Callback inokved when the auth_token is fetched """ matches = ACCESS_TOKEN_REGEX.search(response.body) if matches: access_token = matches.group(1) client = httpclient.AsyncHTTPClient() # lambda is effectively a function factory for us client.fetch(API["profile"] % access_token, lambda response: self.on_profile_fetch(response, access_token)) def on_profile_fetch(self, response, access_token): """ Callback invoked when we have fetched the user's profile """ profile = json.loads(response.body) profile["access_token"] = access_token profile_id = db.save_profile(profile) self.set_secure_cookie("user_id", str(profile_id)) self.redirect("/") # implictly calls self.finish()

Friday, June 10, 2011

Known gotchas

• Immaturity of frameworks, tools

• Tornado templating errors result in somewhat useless stack traces

• MongoDB nuances

• Tornado community fairly small

Friday, June 10, 2011

Questions?

• Ikai Lan - @ikai on Twitter

• Github: https://github.com/ikai

• Google Profile: https://profiles.google.com/ikai.lan/about

• Thank you for having me at Pycon!

Friday, June 10, 2011

Attributions

• Slide 6: Kirstin Jennings - “big smile!” http://www.flickr.com/photos/methyl_lives/2973265796/

• Slide 8: Tony Blay - “I am a bird now!” http://www.flickr.com/photos/toniblay/59415205/

• Slide 12: Antonio Sofi - “whirlwind” http://www.flickr.com/photos/webgol/36546734

• Slide 17: Tornado Web documentation - http://www.tornadoweb.org/documentation

• Slide 18: Christine of Robot Brainz - “Cassette” http://www.flickr.com/photos/robotbrainz/2786347158/

• Slide 23: Samuel Stroube - “Play-Doh on a Picnic Table” http://www.flickr.com/photos/samoube/203586664

Friday, June 10, 2011