Date post: | 08-May-2015 |
Category: |
Technology |
Upload: | ricardo-banffy |
View: | 456 times |
Download: | 0 times |
Boredom comes to those Boredom comes to those who waitwho waitAsynchronous calls to Datastore PlusAsynchronous calls to Datastore Plus
What you should knowWhat you should know
•toplevels/taskletstoplevels/tasklets
•futuresfutures
•yield (you are going to build lots of generators)yield (you are going to build lots of generators)
•you'll use yield instead of return (most of the time) you'll use yield instead of return (most of the time)
•don't think about threadsdon't think about threads
•..._async..._async
•the query API changesthe query API changes
•an insanely great use for comparator overridesan insanely great use for comparator overrides
How it used to be doneHow it used to be done
class Fact(model.Model):class Fact(model.Model): text = model.TextProperty() text = model.TextProperty() rating = model.FloatProperty(default = 400.)rating = model.FloatProperty(default = 400.) random_index = model.ComputedProperty( random_index = model.ComputedProperty(
lambda self : random.randint(0,lambda self : random.randint(0, sys.maxint))sys.maxint))
(...)(...)
for i in range(10):for i in range(10): Fact(text = 'Fact %d' % i).put() Fact(text = 'Fact %d' % i).put()
Digression: _ah/statsDigression: _ah/stats
•A good way to understand the performance of your appsA good way to understand the performance of your apps
•If you are doing something wrong, it'll become obviousIf you are doing something wrong, it'll become obvious
Easy to enable on app.yaml:Easy to enable on app.yaml:
builtins:builtins:(...)(...)- appstats: on- appstats: on
How the first example How the first example behaves locallybehaves locally
… … and on the serverand on the server
The asynchronous wayThe asynchronous way
futures = []futures = []for i in range(10):for i in range(10): futures.append(Fact(text = 'Fact %d' % \ futures.append(Fact(text = 'Fact %d' % \
i).put_async())i).put_async())[ f.get_result() for f in futures ][ f.get_result() for f in futures ]
Gives the opportunity to aggregate puts into one large datastore Gives the opportunity to aggregate puts into one large datastore operation (and we don't have to worry about it)operation (and we don't have to worry about it)
_ah/stats_ah/stats
On the serverOn the server
Better: toplevel/taskletBetter: toplevel/tasklet
(decorating something – usually a request handler – that will call the following tasklet)(decorating something – usually a request handler – that will call the following tasklet)
def init_facts()def init_facts()
futures = []futures = [] for i in range(10): for i in range(10): futures.append(Fact(text = 'Fact %d' % \ futures.append(Fact(text = 'Fact %d' % \
i).put_async())i).put_async()) yield futures yield futures
Yield allows the toplevel event loop to manage other generators making asynchronous callsYield allows the toplevel event loop to manage other generators making asynchronous calls
… … even bettereven better
(decorating your handler)(decorating your handler)
def init_facts()def init_facts()
Futures = []Futures = []
for i in range(10):for i in range(10):
futures.append(Fact(text = 'Fact %d' \futures.append(Fact(text = 'Fact %d' \
% i).put_async())% i).put_async())
raise tasklets.Return(futures)raise tasklets.Return(futures)
Because it's considered polite to raise things when a generator has nothing else to generateBecause it's considered polite to raise things when a generator has nothing else to generate
ab -n 10000 -c 50 (synchronous)ab -n 10000 -c 50 (synchronous)
Connection Times (ms)Connection Times (ms) min mean[+/-sd] median max min mean[+/-sd] median max Connect: 140 159 69.1 145 976Connect: 140 159 69.1 145 976Processing: 338 7408 5452.2 6231 46247 Processing: 338 7408 5452.2 6231 46247 Waiting: 338 7407 5452.2 6230 46247 Waiting: 338 7407 5452.2 6230 46247 Total: Total: 482 7567 5442.4 6377 46401 482 7567 5442.4 6377 46401
Percentage of the requests served within a certain time (ms)Percentage of the requests served within a certain time (ms) 50% 6377 50% 6377 66% 8540 66% 8540 75% 10131 75% 10131 80% 11068 80% 11068 90% 13419 90% 13419 95% 16077 95% 16077 98% 23883 98% 23883 99% 30173 99% 30173 100% 46401 (longest request) 100% 46401 (longest request)
ab -n 10000 -c 50 (asynchronous)ab -n 10000 -c 50 (asynchronous)
Connection Times (ms)Connection Times (ms) min mean[+/-sd] median max min mean[+/-sd] median max Connect: 140 669 1375.6 151 21193Connect: 140 669 1375.6 151 21193Processing: 189 338 300.0 256 15320Processing: 189 338 300.0 256 15320Waiting: 189 335 243.7 255 4143Waiting: 189 335 243.7 255 4143Total: Total: 332 1007 1407.6 438 21450332 1007 1407.6 438 21450
Percentage of the requests served within a certain time (ms)Percentage of the requests served within a certain time (ms) 50% 438 50% 438 66% 565 66% 565 75% 732 75% 732 80% 1272 80% 1272 90% 3372 90% 3372 95% 3456 95% 3456 98% 3762 98% 3762 99% 9366 99% 9366 100% 21450 (longest request) 100% 21450 (longest request)
What's the difference?What's the difference?
WrongWrong
for f in Fact.query():for f in Fact.query():
f.rating = random.normalvariate(400, 20)f.rating = random.normalvariate(400, 20)
f.put() f.put()
Why?Why?
Right: map_asyncRight: map_async
@[email protected] randomize_rating(f):def randomize_rating(f): f.rating = random.normalvariate(400, 20) f.rating = random.normalvariate(400, 20) raise tasklets.Return(f.put_async()) raise tasklets.Return(f.put_async())
@[email protected] randomize_all():def randomize_all(): Fact.query().map_async(randomize_rating) Fact.query().map_async(randomize_rating)
Right: map_asyncRight: map_async
What else should I know?What else should I know?
•context and its event loopcontext and its event loop
•cachescaches
•new datatypesnew datatypes
•new names for old typesnew names for old types
•repeated = True repeated = True
•StructuredProperty, LocalStructuredPropertyStructuredProperty, LocalStructuredProperty
•compresscompress
•shorter response times and more efficient instance usageshorter response times and more efficient instance usage
Where do I find it?Where do I find it?
•official buildsofficial builds
•http://code.google.com/p/appengine-ndb-experiment/downloads/listhttp://code.google.com/p/appengine-ndb-experiment/downloads/list
•"Bleeding" edge "Bleeding" edge
•hg clone https://code.google.com/p/appengine-ndb-experiment/hg clone https://code.google.com/p/appengine-ndb-experiment/
•version 0.7 released yesterdayversion 0.7 released yesterday
To know moreTo know more
•documentation:documentation:
http://code.google.com/p/appengine-ndb-experiment/http://code.google.com/p/appengine-ndb-experiment/
•Google group:Google group:
http://groups.google.com/group/appengine-ndb-discuss/http://groups.google.com/group/appengine-ndb-discuss/
ThanksThanks
Thanks to the fine people who hang out on the appengine-ndb-discuss Thanks to the fine people who hang out on the appengine-ndb-discuss group, in special to Guido and Vladimir, whose suggestions pointed me on group, in special to Guido and Vladimir, whose suggestions pointed me on the right direction.the right direction.