Code Lab - Django

7/27/2019 Code Lab - Django

1/132

django

Code Lab

James Bennett Adrian Holovaty Jacob Kaplan-Moss

PyCon 2008Thursday, March 13, 2008


2/132

Introductions


3/132

James BennettDeveloper, Lawrence Journal-World

Release Manager, Django

Author, Practical Django Projects

Professional Asshole


4/132

Adrian HolovatyFounder, EveryBlock

BDFL, Django

Author, The Definitive Guide to Django

YouTube Phenomenon


5/132

Jacob Kaplan-MossPartner, Whiskey Media

BDFL, Django

Author, The Definitive Guide to Django


6/132

Schedule

6:20 - 7:40 Code lab, part 17:40 - 8:00 Break

8:00 - 9:00 Code lab, part 2

9:00 - 9:30 Stump the Chumps!


7/132

Code lab, part 1

Pim Van Heuven: URL design

Justin Lilly: unit testing and TDD

Richard House: model design

Peter Herndon: searching, optimization

J. Clifford Dyer: next/previous links

Dave Lowe: when not to use the admin


8/132

Code lab, part 2

Bob Haugen: preparing for deployment

Wiley Kestner: signals

Eric St-Jean: REST APIs

Sean OConnor: order_by(?)

Honza Krl: many questions, many answers?


9/132

Pim van HeuvenURL design


10/132

The case of the monsterURLConf


11/132

urls.py

2032 lines

600+ URL patterns

94 dictionaries of keyword arguments


12/132

Worst-case matching

Every time you try to match a regex and

fail, it takes timeWorst case is a 404 which matchesnothing

Has to fail over 600 regexes before it canreturn


13/132

Lets break things up


14/132

urlpatterns = urlpatterns + patterns('d3a.ajax.views',

(r'^pricegroup/json/validate/$', 'validate_generic', {'form' : PricegroupForm } ),

)

urlpatterns = urlpatterns + patterns('ledger.views',

(r'^pricegroup/add/$', 'create_object', pricegroup_dict),

(r'^pricegroup/print_or_email/(?P\w+)/(?P\w*)/(?P\d+)/$',

'print_or_email', pricegroup_dict),

(r'^pricegroup/(?P\d+)/$', 'update_object', pricegroup_dict),

(r'^pricegroup/(?P\d+)/delete/$', 'delete_object', dict(pricegroup_dict,

post_delete_redirect='../../')),

)

urlpatterns = urlpatterns + patterns('ledger.views',

(r'^partsvendor/$', 'object_list', partsvendor_dict),

(r'^partsvendor/json_table/$', 'json_table', partsvendor_dict),

)

urlpatterns = urlpatterns + patterns('d3a.ajax.views',

(r'^partsvendor/json/validate/$', 'validate_generic', {'form' : PartsvendorForm } ),

)


15/132

These are alreadylogically separate


16/132

Split into multiple files,use include()


17/132

Worst-case matching

Break the URLConf up into logical bits,several levels deep

Worst case will be a couple dozen fails


18/132

And now for somethingcompletely different...


19/132

Forms with extraparameters


20/132

Use cases

You want to pass an existing object to a

form and derive initial data from it

You want to pass a user into a form andhave the saved object be related to it

etc., etc.


21/132

Getting initial data

For each field on the object which willhave a field in the form, get the value

Build a dictionary of field name -> field

value mappings


22/132

Django can do this

django.newforms.models.model_to_dict()

Takes an object, and optionally lists offields to include/exclude


23/132

obj = SomeModel.objects.get(pk=5)

initial_data = model_to_dict(obj)

form = MyForm(initial=initial_data)


24/132

How about extraparameters?


25/132

Simple pattern

Override __init__()

Take any extra arguments you want, plus*args and **kwargs

Then call superclass __init__(), passing*args and **kwargs


26/132

class MyForm(forms.Form):

def __init__(self, user, *args, **kwargs)

self.user = user

super(MyForm, self).__init__(*args,

**kwargs)


27/132

super() is important;dont leave it out


28/132

Other tricks

After super() call, mess around withself.fields

Its just a dictionary of field name -> field

objects


29/132

Justin LillyUnit Testing and TDD


30/132

Tests are theProgrammers stone,

transmuting fear intoboredom.

Kent Beck


31/132

Hardcore TDD


32/132

I dont do test drivendevelopment. I do stupiditydriven testing I wait until

I do something stupid, andthen write tests to avoiddoing it again.

Titus Brown


33/132

Whatever happens, dont letyour test suite break thinking,Ill go back and fix this later.


34/132

Testing Django

Doctests

Unit testsFixtures

Test client

Email capture


35/132

Testing Django

Doctests

Unit testsFixtures

Test client

Email capture


36/132

"""

>>> from django.test import Client>>> c = Client()

>>> r = c.get("/calendar/")

>>> r.status_code200

>>> r.context[1].date

datetime.datetime(...)

"""


37/132


38/132

$ ./manage.py runtests

Ran 1 test in 0.019s

OK


39/132

Testing Django

Doctests

Unit testsFixtures

Test client

Email capture


40/132

Unit tests


41/132

from datetime import datetime

from django.test import TestCase

from django.template import Template, Context

class ScheduleTestCase(TestCase):

def setUp(self):

template.add_to_builtins("schedule.templatetags.schedule_cal")

def render(t, **c):return Template(t).render(Context(c))

def testTag(self):t = "{% schedule_cal date items %}"

c = {

"date": datetime(2007, 1),"cal_items": [ ... ]

}

r = self.render(t, c)

self.assert_("Jan 2007" in r)


42/132

Fixtures


43/132

class ScheduleTestCase(TestCase):

fixtures = ["project_testdata"]

...


44/132

model: projects.project

pk: 1fields:

name: Foo

due: 20071001 11:59

model: schedule.meeting

pk: 1

fields:

projects: [1]

date: 20070903 20:00

[


45/132

[

{

"model": "projects.project",

"pk": 1,

"fields": {

"name": "Foo",

"due": "20071001 11:59"

}

},{

"model": "schedule.meeting",

"pk": 1,

"fields": {

"projects": [1],

"date": "20070903 20:00",

}

}

]


46/132

Foo

20071001 11:59

2007093 20:00


47/132

Doctests or Unit Tests?


48/132

Richard HouseModel design


49/132

We just need one more

field...


50/132

send_confirmatory_email = models.BooleanField(default=True,

null=True, )telephone = models.BooleanField(default=False, null=True, )text_message = models.BooleanField(default=False, null=True, )

address = models.BooleanField(default=False, null=True, )

car_reg = models.BooleanField(default=False, null=True, )

nationality = models.BooleanField(default=False, null=True, )

dietary = models.BooleanField(default=False, null=True, )dinner = models.BooleanField(default=False, null=True, )

company = models.BooleanField(default=False, null=True, )

ni_number = models.BooleanField(default=False, null=True, )

date_of_birth = models.BooleanField(default=False, null=True, )

place_of_birth = models.BooleanField(default=False, null=True, )guests = models.BooleanField(default=False, null=True, )


51/132

Represent this information relationally

Dont put all the fields on all the events

An easier way


52/132

Two pairs of models


53/132

First pair

EventType encapsulates a type of event

EventTypeOption will represent a piece ofinformation


54/132

class EventType(models.Model):

name = models.CharField(max_length=255)

class EventTypeOption(models.Model):

name = models.CharField(max_length=255)field_label = models.CharField(max_length=255)

event_type = models.ForeignKey(EventType,

related_name=option)


55/132

Event encapsulates an event, and has allthe fields common to all events

EventOption will be a piece of informationfor a specific event

Second pair


56/132

class Event(models.Model):

# core fields here...

event_type = models.ForeignKey(EventType)

class EventOption(models.Model):

event = models.ForeignKey(Event, related_name=option)

option = models.ForeignKey(EventTypeOption)

value = models.BooleanField()


57/132

Lets build a form


58/132

def __init__(self, event_type, *args, **kwargs):

self.event_type = event_type

super(EventForm, self).__init__(*args, **kwargs)for option in self.event_type.option_set.all():

self.fields[option.name] = forms.BooleanField(label=option.field_label)


59/132

Lets save an event


60/132

def save(self):

# Construct an Event object and save it first...

for option in self.event_type.option_set.all():if option.name in self.cleaned_data:

new_event.option_set.create(option=option,

value=self.cleaned_data[option.name])


61/132

Lets display an event


62/132

{# show normal fields first #}

{% for event_option in event.option_set.all %}

{{ event_option.option.name }}:{{ event_option.value }}

{% endfor %}


63/132

Now you just add

EventTypeOptions foreach new request


64/132

No more schemachanges


65/132

Yay!


66/132

Peter HerndonSearching: query optimization, executing raw SQL, andchoosing the right tool for the job.


67/132

The problem


68/132

/people/jacobkaplanmoss/


69/132

Person.objects.filter(slug="jacobkaplanmoss")


70/132

/documents/search/?keywords=foo


71/132

QuerySet?

def search_all(search_data):q = Q()key_documents = Noneresults = None


72/132

results None

if search_data['author'] and not search_data['author'] == 'Last Name, First':# parse author names# ex: 'Norton, L' or 'Norton, L; Begg, C'authors = search_data['author'].rstrip()authors = authors.rstrip(';')authors = authors.split(';')# ex: ['Norton, L'] or ['Norton, L', ' Begg, C']authors = map(string.strip, authors)document_id_lists = []intersection_set = set()

for author in authors:document_ids = []pubs = []name_parts = author.split(', ')if len(name_parts) == 2:

lname, fname = name_partsemployees = Employee.objects.filter(last_name__iexact=lname,

first_name__iexact=fname)else:

lname = name_parts[0]employees = Employee.objects.filter(last_name__iexact=lname)

for employee in employees:if pubs:

pubs = pubs | employee.publication_set.all()else:

pubs = employee.publication_set.all()for pub in pubs:

document_ids.append(pub.document.id)document_id_lists.append(document_ids)for document_id_list in document_id_lists:

if document_id_list:document_id_set = set(document_id_list)if intersection_set:

intersection_set =intersection_set.intersection(document_id_set)

else:intersection_set = document_id_set

q = q & Q(id__in=intersection_set)

if search_data['journal'] and not search_data['journal'] == 'Ex: Blood':journals = search_data['journal'].split(';')journals = map(string.strip, journals)print "journals:", journalsfor journal in journals:

q = q & Q(source__name__iexact=journal)

if search_data['dmt'] and not search_data['dmt'] == '0':q = q & Q(dmt__id__exact=search_data['dmt'])

if (search_data['year_start'] and not search_data['year_start'] == 'BLANK') or

(search_data['year_end'] and not search_data['year_end'] == 'BLANK'):start = search_data['year_start']end = search_data['year_end']if not start or start == 'BLANK':

start = datetime.datetime.now().yearif not end or end == 'BLANK':

end = datetime.datetime.now().year

q = q & Q(publish_year__range=(start, end))

if search_data['doc_type']:

if isinstance(search_data['doc_type'], list):q = q & Q(document_type__in=search_data['doc_type'])

else:q = q & Q(document_type__istartswith=search_data['doc_type'])

if search_data['keywords'] and not search_data['keywords'] == 'Ex: melanoma':

keywords = search_data['keywords'].split()title_documents = Noneabstract_documents = Nonekeyword_documents = None

for word in keywords:title_docs = Document.objects.filter(title__icontains=word)abstract_docs = Document.objects.filter(abstract__icontains=word)kwords = Keyword.objects.filter(term__icontains=word)keyword_doc_ids = [kw.document.id for kw in kwords]keyword_docs = Document.objects.filter(id__in=keyword_doc_ids)if title_documents:

title_documents = title_docs & title_documentselse:

title_documents = title_docsif abstract_documents:

abstract_documents = abstract_docs & abstract_documentselse:

abstract_documents = abstract_docsif keyword_documents:

keyword_documents = keyword_documents & keyword_docs

else:keyword_documents = keyword_docs

if title_documents:if key_documents:

key_documents = key_documents | title_documentselse:

key_documents = title_documents

if abstract_documents:if key_documents:

key_documents = key_documents | abstract_documentselse:

key_documents = abstract_documentsif keyword_documents:

if key_documents:key_documents = key_documents | keyword_documents

else:key_documents = keyword_documents

if key_documents:results = Document.objects.filter(q).select_related().order_by('

publish_year') & key_documentselif (hasattr(q, 'kwargs') and q.kwargs) or (hasattr(q, 'args') and q.args):

results = Document.objects.filter(q).select_related().order_by('publish_year')

else:results = []

return results


73/132

if search_data['dmt'] and not search_data['dmt'] == '0':

q = q & Q(dmt__id__exact=search_data['dmt'])

...

results = Document.objects.filter(q)


74/132


75/132

Why so slow?

Opaque data structures

Cloning QuerySets

Inefficient SQL

Wrong data structure


76/132

Raw SQL


77/132

from django import db

sql = "SELECT FROM WHERE end = %s "

params = [end_date, ]

cursor = db.connection.cursor()

cursor.execute(sql, params)

for row in cursor:

...


78/132


79/132

Linus Torvalds

Bad programmers worryabout the code. Good

programmers worry aboutdata structures....


80/132

Search is asolved problem


81/132

Tsearch2 MySQL FULLTEXT


82/132

http://code.google.com/p/djapian/

http://code.google.com/p/django-sphinx/

http://code.google.com/p/djangosearch
http://code.google.com/p/django-sphinx/http://code.google.com/p/django-sphinx/http://code.google.com/p/django-sphinx/http://code.google.com/p/djapian/http://code.google.com/p/djapian/


83/132

from djangosearch.indexer import ModelIndex

class Document(models.Model):

index = ModelIndex()


84/132

results = Document.index.search(query)


85/132

J. Clifford DyerHandling previous/next links


86/132

Dave LoweWhen not to use the admin


87/132

Break


88/132

Bob HaugenPrepping for deployment


89/132

Coding for deployment


90/132

if settings.DEBUG:

urlpatterns += patterns('',

(r'^site_media/(?P.*)$',

'django.views.static.serve',

{'document_root': '...'}),

)


91/132

HOME = os.path.abspath(os.path.dirname(__file__))

TEMPLATE_DIRS = (HOME + "templates/",

)


92/132

HOME = os.path.abspath(os.path.dirname(__file__))

TEMPLATE_DIRS = (os.path.join(HOME, "templates/"),

)


93/132

return HttpResponseRedirect(

'%s/%s/' % ('order', new_order.id))


94/132

return HttpResponseRedirect(

urlresolvers.reverse(order, args=[new_order.id]))


95/132

raise Http404


96/132

raise Http404("Invalid order ID")

D l D l


97/132

Develop Deploymanage.py runserver Apache + mod_python

mod_wsgi

SQLite PostgreSQL/MySQL

DEBUG = True DEBUG = False

views.static.serve lighttpd

Django Django + other stuff*


98/132

* Other stuff


99/132

Perlbalreverse proxy load balancer and web serverhttp://danga.com/perlbal/

Wh ?
http://danga.com/perlbal/http://danga.com/perlbal/


100/132

Why?

SpoonfeedingAbstraction of hardware resources

Load balancing


101/132

http://danga.com/perlbal/

http://code.sixapart.com/svn/perlbal/trunk/doc/

http://lists.danga.com/mailman/listinfo/perlbal

HOWTO

CREATE POOL ljservers

set nodefile = /etc/perlbal/nodes/ljworld
http://lists.danga.com/mailman/listinfo/perlbalhttp://lists.danga.com/mailman/listinfo/perlbalhttp://lists.danga.com/mailman/listinfo/perlbalhttp://code.sixapart.com/svn/perlbal/trunk/doc/http://code.sixapart.com/svn/perlbal/trunk/doc/http://danga.com/perlbal/http://danga.com/perlbal/


102/132

set nodefile = /etc/perlbal/nodes/ljworld

CREATE SERVICE ljworld

SET role = reverse_proxySET pool = ljservers

SET buffer_size = 80k

ENABLE ljworld

CREATE SERVICE vhosts

SET listen = 0.0.0.0:80

SET role = selectorSET plugins = vhosts

SET persist_client = on

VHOST ljworld.com = ljworldVHOST *.ljworld.com = ljworld

ENABLE vhosts


103/132

Memcachedhigh-performance, distributed memory object caching systemhttp://danga.com/memcached/

Wh ?
http://danga.com/memcached/http://danga.com/memcached/


104/132

Why?

Anything less would be uncivilized.

Seriously, just use it.


105/132

root@example# memcached d m 2048 l 10.0.0.40 p 11211


106/132

CACHE_BACKEND = "memcached://10.0.0.40:11211/"


107/132

Wiley KestnerSignals


108/132

B k d


109/132

Background

Sometimes called the Observer patternin communities with design-pattern focus

Usually handled with some sort ofdispatcher


110/132

Django has a dispatcher

Dispatcher 101


111/132

Some piece of code sends a signal

A signal is just a pre-defined Pythonobject

Django defines a bunch of useful signalsfor you

django.dispatch.dispatcher.send()

Dispatcher 101

Dispatcher 101


112/132

Dispatcher 101

Functions use the dispatcher to connectto a signal

Once connected, that function will becalled whenever something else sendsthat signal

django.dispatch.dispatcher.connect()


113/132

from django.dispatch import dispatcher

from django.db.models import signals

def save_callback():

print Something just got saved

dispatcher.connect(save_callback,

signal=signals.post_save)

The tricky bit


114/132

Your call to dispatcher.connect() has tobe executed

Good place to put it: models.py

The tricky bit


115/132

Lets write a simplelistener


116/132

class NotificationManager(models.Manager):

def notify_comment(self, sender, instance):for n in self.filter(object_id=instance.object_id,

content_type__pk=instance.content_type_id):

n.notify()

class Notification(models.Model):# ...

objects = NotificationManager()

def notify(self):

# ...


117/132

The secret sauce


118/132

from django.db.models import signals

from django.dispatch import dispatcher

from notifications.models import Notification

from django.contrib.comments.models import Comment

dispatcher.connect(Notification.objects.notify_comment,

sender=Comment,

signal=signals.post_save)

Extending it


119/132

Extending it

Use the sender argument to control themodel you listen to

Or omit it to listen to everything


120/132

About genericlisteners

Should it be generic?


121/132

Should it be generic?

Well, itd be more reusableBut Django already provides a generic wayto do this: the dispatcher


122/132

Coupling site-specific

functions to a specificsite isnt wrong


123/132

Eric St-JeanREST APIs


124/132

Terminology


125/132

APIApplication Programming Interface


126/132

RESTRepresentational State Transfer

UR DOIN IT RITE


127/132

UR DOIN IT RITE

Proper URIs

Correct use of HTTP

Valid, semantically-rich formats

Resources, not methods


128/132


129/132

class Something(Model):

...

@ws.web_method(scope='object', return_type='object', method='POST')

@ws.web_arg(name='body', type='string')

def reply(self, body, _user):

@ws.web_method(scope='type', return_type='object', method='POST')

@ws.web_arg(name='content_object', type='object')

@ws.web_arg(name='user', type='object')

@ws.web_arg(name='subject', type='string')

@ws.web_arg(name='message', type='string')

def recommend(self, request, input_data, queryset):


130/132

Sean OConnororder_by(?)


131/132

Honza KrlMany questionsMany answers?


132/132

Stump

theChumps!

Date post:	02-Apr-2018
Category:	Documents
Upload:	roulis-paparoulis
View:	233 times
Download:	0 times

Code Lab - Django

Documents