Elegant Solutions For Everyday Python ProblemsNina Zakharenko@nnjabit.ly/elegant-python-caℹ There are links in these slides. Follow along ^

This talk is for you if:— You're an intermediate python programmer— You're coming to python from another language— You want to learn about fancy features like: magic

methods, iterators, decorators, and context managers

slides: bit.ly/elegant-python-ca

@nnja

Whatiselegant code?@nnja

How do we make code elegant?We pick the right tool for the job!

Resources for converting from Python 2 -> 3

Beauty isin the eye ofthe beholder

magicmethods

image source

You're used to implementing __str__ and __repr__ --but there's a whole other world of powerful magic methods!

By implementing a few straightforward methods,you can make your objects behave like built-ins such as:

— numbers— lists— dictionaries— and more...

@nnja

class Money:

currency_rates = { '$': 1, '€': 0.88, }

def __init__(self, symbol, amount): self.symbol = symbol self.amount = amount

def __repr__(self): return '%s%.2f' % (self.symbol, self.amount)

@nnja

class Money:

currency_rates = { '$': 1, '€': 0.88, }

def __init__(self, symbol, amount): self.symbol = symbol self.amount = amount

def __repr__(self): return '%s%.2f' % (self.symbol, self.amount)

@nnja

class Money:

# defined currency_rates, __init__, and repr above...

def convert(self, other): """Convert other amount to our currency""" new_amount = ( other.amount / self.currency_rates[other.symbol] * self.currency_rates[self.symbol]) return Money(self.symbol, new_amount)

@nnja

__repr__ in action

>>> soda_cost = Money('$', 5.25)

>>> soda_cost $5.25

>>> pizza_cost = Money('€', 7.99)

>>> pizza_cost €7.99

@nnja

class Money:

def __add__(self, other): """ Add 2 Money instances using '+' """ new_amount = self.amount + self.convert(other).amount return Money(self.symbol, new_amount)

@nnja

>>> soda_cost = Money('$', 5.25)

>>> pizza_cost = Money('€', 7.99)

>>> soda_cost + pizza_cost $14.33

More on Magic Methods: Dive into Python3 - Special Method Names

>>> soda_cost = Money('$', 5.25)

>>> pizza_cost = Money('€', 7.99)

>>> soda_cost + pizza_cost $14.33

>>> pizza_cost + soda_cost €12.61

More on Magic Methods: Dive into Python3 - Special Method Names

@nnja

some magic methods map to built-in functions

class Alphabet: letters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'

def __len__(self): return len(self.letters)

>>> my_alphabet = Alphabet()>>> len(my_alphabet) 26

@nnja

customiterators

image source

Making classes iterable— In order to be iterable, a class needs to implement

__iter__()— __iter__() must return an iterator— In order to be an iterator a class needs to implement

__next__() which must raise StopIteration when there are no more items to return

or next() in python2

^ can be confusing at first, but remember these guidelines for making classes Great explanation of iterable vs. iterator vs. generator

class IterableServer:

services = [ {'active': False, 'protocol': 'ftp', 'port': 21}, {'active': True, 'protocol': 'ssh', 'port': 22}, {'active': True, 'protocol': 'http', 'port': 21}, ] def __init__(self): self.current_pos = 0 def __iter__(self): # can return self, because __next__ implemented return self def __next__(self): while self.current_pos < len(self.services): service = self.services[self.current_pos] self.current_pos += 1 if service['active']: return service['protocol'], service['port'] raise StopIteration next = __next__ # optional python2 compatibility

@nnja

>>> for protocol, port in IterableServer(): print('service %s is running on port %d' % (protocol, port))

service ssh is running on port 22service http is running on port 21

... not bad@nnja

tip: use a generatorwhen your iterator doesn't need to maintain a lot of state

@nnja

class Server:

services = [ {'active': False, 'protocol': 'ftp', 'port': 21}, {'active': True, 'protocol': 'ssh', 'port': 22}, {'active': True, 'protocol': 'http', 'port': 21}, ]

def __iter__(self): for service in self.services: if service['active']: yield service['protocol'], service['port']

@nnja

class Server:

services = [ {'active': False, 'protocol': 'ftp', 'port': 21}, {'active': True, 'protocol': 'ssh', 'port': 22}, {'active': True, 'protocol': 'http', 'port': 21}, ]

def __iter__(self): for service in self.services: if service['active']: yield service['protocol'], service['port']

@nnja

Why does this work?use single parenthesis ( ) to create a generator comprehension^ technically, a generator expression but I like this term better, and so does Ned Batchelder

>>> my_gen = (num for num in range(1))>>> my_gen <generator object <genexpr> at 0x107581bf8>

@nnja

An iterator must implement __next__()

>>> next(my_gen) # remember __len__() mapped to built-in len() 0

and raise StopIteration when there are no more elements

>>> next(my_gen)... StopIteration Traceback (most recent call last)

For more tools for working with iterators, check out itertools

✨Method

✨

✨

Magic

✨

@nnja

alias methods

class Word:

def __init__(self, word): self.word = word

def __repr__(self): return self.word

def __add__(self, other_word): return Word('%s %s' % (self.word, other_word))

# Add an alias from method __add__ to the method concat concat = __add__

@nnja

When we add an alias from __add__ to concat because methods are just objects

>>> # remember, concat = __add__>>> first_name = Word('Max')>>> last_name = Word('Smith')

>>> first_name + last_name Max Smith

>>> first_name.concat(last_name) Max Smith

>>> Word.__add__ == Word.concat True@nnja

Dog class

>>> class Dog: sound = 'Bark' def speak(self): print(self.sound + '!', self.sound + '!')

>>> my_dog = Dog()>>> my_dog.speak() Bark! Bark!

read the docs

getattr(object, name, default)

>>> class Dog: sound = 'Bark' def speak(self): print(self.sound + '!', self.sound + '!')

>>> my_dog = Dog()>>> my_dog.speak() Bark! Bark!

>>> getattr(my_dog, 'speak') <bound method Dog.speak of <__main__.Dog object at 0x10b145f28>>

>>> speak_method = getattr(my_dog, 'speak')>>> speak_method() Bark! Bark!

read the docs

getattr(object, name, default)

>>> class Dog: sound = 'Bark' def speak(self): print(self.sound + '!', self.sound + '!')

>>> my_dog = Dog()>>> my_dog.speak() Bark! Bark!

>>> getattr(my_dog, 'speak') <bound method Dog.speak of <__main__.Dog object at 0x10b145f28>>

>>> speak_method = getattr(my_dog, 'speak')>>> speak_method() Bark! Bark!

read the docs

Example: command line tool with dynamic commands

class Operations: def say_hi(self, name): print('Hello,', name)

def say_bye(self, name): print ('Goodbye,', name)

def default(self, arg): print ('This operation is not supported.')

if __name__ == '__main__': operations = Operations() # let's assume error handling command, argument = input('> ').split() getattr(operations, command, operations.default)(argument)

read the docs

Output

$ python getattr.py

> say_hi Nina

Hello, Nina

> blah blah

This operation is not supported.✨

additional reading - inverse of getattr() is setattr()

functool.partial(func, *args, **kwargs)

— Return a new partial object which behaves like func called with args & kwargs

— if more args are passed in, they are appended to args— if more keyword arguments are passed in, they extend

and override kwargs

read the docs on partials

functool.partial(func, *args, **kwargs)

>>> from functools import partial>>> basetwo = partial(int, base=2)>>> basetwo functools.partial(<class 'int'>, base=2)>>> basetwo('10010') 18

read the docs on partials

library I !

: github.com/jpaugh/agithub

agithub is a (badly named) REST API client with transparent syntax which facilitates rapid prototyping — on any REST API!

— Implemented in 400 lines.— Add support for any REST API in ~30 lines of code.— agithub knows everything it needs to about protocol

(REST, HTTP, TCP), but assumes nothing about your upstream API.

@nnja

define endpoint url & other connection properties

class GitHub(API):

def __init__(self, token=None, *args, **kwargs):

props = ConnectionProperties(

api_url = kwargs.pop('api_url', 'api.github.com'))

self.setClient(Client(*args, **kwargs))

self.setConnectionProperties(props)

then, start using the API!

>>> gh = GitHub('token')

>>> status, data = gh.user.repos.get(visibility='public', sort='created')

>>> # ^ Maps to GET /user/repos

>>> data

... ['tweeter', 'snipey', '...']

github.com/jpaugh/agithub

black magic!but, how?...

!

@nnja

class API: def __getattr__(self, key): return IncompleteRequest(self.client).__getattr__(key) __getitem__ = __getattr__

class IncompleteRequest: def __getattr__(self, key): if key in self.client.http_methods: htmlMethod = getattr(self.client, key) return partial(htmlMethod, url=self.url) else: self.url += '/' + str(key) return self __getitem__ = __getattr__

class Client: http_methods = ('get') # ...

def get(self, url, headers={}, **params): return self.request('GET', url, None, headers)

github.com/jpaugh/agithub source: base.py

class API: def __getattr__(self, key): return IncompleteRequest(self.client).__getattr__(key) __getitem__ = __getattr__

class IncompleteRequest: def __getattr__(self, key): if key in self.client.http_methods: htmlMethod = getattr(self.client, key) return partial(htmlMethod, url=self.url) else: self.url += '/' + str(key) return self __getitem__ = __getattr__

class Client: http_methods = ('get') # ...

def get(self, url, headers={}, **params): return self.request('GET', url, None, headers)

github.com/jpaugh/agithub source: base.py

class API: def __getattr__(self, key): return IncompleteRequest(self.client).__getattr__(key) __getitem__ = __getattr__

class IncompleteRequest: def __getattr__(self, key): if key in self.client.http_methods: htmlMethod = getattr(self.client, key) return partial(htmlMethod, url=self.url) else: self.url += '/' + str(key) return self __getitem__ = __getattr__

class Client: http_methods = ('get') # ...

def get(self, url, headers={}, **params): return self.request('GET', url, None, headers)

github.com/jpaugh/agithub source: base.py

given a non-existant path:

>>> status, data = this.path.doesnt.exist.get()>>> status... 404

& because __getitem__ is aliased to __getattr__:

>>> owner, repo = 'nnja', 'tweeter'>>> status, data = gh.repos[owner][repo].pulls.get()>>> # ^ Maps to GET /repos/nnja/tweeter/pulls>>> data.... # {....}

github.com/jpaugh/agithub

Context Managers& new in python 3: async context managers

When should I use one?Need to perform an action before and/or after an operation.

Common scenarios:

— Closing a resource after you're done with it (file, network connection)

— Perform cleanup before/after a function call

@nnja

Example Problem: Feature Flags

Turn features of your application on and off easily.

Uses of feature flags:

— A/B Testing— Rolling Releases — Show Beta version to users opted-in to Beta Testing

Program

More on Feature Flags

Example - FeatureFlags Class

class FeatureFlags: """ Example class which stores Feature Flags and their state. """

SHOW_BETA = 'Show Beta version of Home Page'

flags = { SHOW_BETA: True }

@classmethod def is_on(cls, name): return cls.flags[name]

@classmethod def toggle(cls, name, on): cls.flags[name] = on

feature_flags = FeatureFlags()

@nnja

How do we temporarily turn features on and off when testing flags?

Want:

with feature_flag(FeatureFlags.SHOW_BETA): assert '/beta' == get_homepage_url()

@nnja

Using Magic Methods __enter__ and __exit__

class feature_flag: """ Implementing a Context Manager using Magic Methods """

def __init__(self, name, on=True): self.name = name self.on = on self.old_value = feature_flags.is_on(name)

def __enter__(self): feature_flags.toggle(self.name, self.on)

def __exit__(self, *args): feature_flags.toggle(self.name, self.old_value)

See: contextlib.contextmanager

The be!er way: using the contextmanager decorator

from contextlib import contextmanager

@contextmanagerdef feature_flag(name, on=True): old_value = feature_flags.is_on(name) feature_flags.toggle(name, on) yield feature_flags.toggle(name, old_value)

See: contextlib.contextmanager

The be!er way: using the contextmanager decorator

from contextlib import contextmanager

@contextmanagerdef feature_flag(name, on=True): """ The easier way to create Context Managers """ old_value = feature_flags.is_on(name) feature_flags.toggle(name, on) # behavior of __enter__() yield feature_flags.toggle(name, old_value) # behavior of __exit__()

See: contextlib.contextmanager

Note: yield?

from contextlib import contextmanager

@contextmanagerdef feature_flag(name, on=True): """ The easier way to create Context Managers """ old_value = feature_flags.is_on(name) feature_flags.toggle(name, on) # behavior of __enter__() yield feature_flags.toggle(name, old_value) # behavior of __exit__()

See: contextlib.contextmanager

either implementation

def get_homepage_url(): """ Method that returns the path of the home page we want to display. """ if feature_flags.is_on(FeatureFlags.SHOW_BETA): return '/beta' else: return '/homepage'

def test_homepage_url_with_context_manager():

with feature_flag(FeatureFlags.SHOW_BETA): # saw the beta homepage... assert get_homepage_url() == '/beta'

with feature_flag(FeatureFlags.SHOW_BETA, on=False): # saw the standard homepage... assert get_homepage_url() == '/homepage'

@nnja

either implementation

def get_homepage_url(): """ Method that returns the path of the home page we want to display. """ if feature_flags.is_on(FeatureFlags.SHOW_BETA): return '/beta' else: return '/homepage'

def test_homepage_url_with_context_manager():

with feature_flag(FeatureFlags.SHOW_BETA): assert get_homepage_url() == '/beta' print('seeing the beta homepage...')

with feature_flag(FeatureFlags.SHOW_BETA, on=False): assert get_homepage_url() == '/homepage' print('seeing the standard homepage...')

@nnja

DecoratorsThe simple explanation:

Syntactic sugar that allows modification of an underlying function.

@nnja

Recap!Decorators:— Wrap a function in another function.— Do something:

— before the call— after the call— with provided arguments— modify the return value or arguments

@nnja

def say_after(hello_function):

def say_nice_to_meet_you(name):

hello_function(name)

print('It was nice to meet you!')

return say_nice_to_meet_you

def hello(name):

print('Hello', name)

>>> hello('Nina')

Hello Nina

>>> say_after(hello)('Nina')

Hello Nina It was nice to meet you!

— say_after(hello) returns the function say_nice_to_meet_you

— then we call say_nice_to_meet_you('Nina')@nnja

def say_after(hello_function):

def say_nice_to_meet_you(name):

hello_function(name)

print('It was nice to meet you!')

return say_nice_to_meet_you

@say_after

def hello(name):

print('Hello', name)

>>> hello('Nina')

Hello Nina It was nice to meet you!

— calling the decorated function hello(name)— is the same as calling an undecorated hello with

say_after(hello)('Nina')

@nnja

closure example

def multiply_by(num): def do_multiplication(x): return x * num return do_multiplication

multiply_by_five = multiply_by(5)

>>> multiply_by_five(4) 20

@nnja

decorators that take arguments

def greeting(argument): def greeting_decorator(greet_function): def greet(name): greet_function(name) print('It was %s to meet you!' % argument) return greet return greeting_decorator

@greeting('bad')def aloha(name): print ('Aloha', name)

@nnja

decorators that take arguments

def say_this_after(argument):

def say_after(hello_function):

def say_after_meeting(name):

hello_function(name)

print('It was %s to meet you' % argument)

return say_after_meeting

return say_after

@say_this_after('bad')

def hello(name):

print('Hello', name)

Is the same as calling this on an undecorated function:

say_after_bad = say_this_after('bad')(hello)say_after_bad('Nina')

@nnja

losing context with a decorator !

def say_bye(func): def wrapper(name): func() print('Bye', name) return wrapper

@say_byedef my_name():""" Say my name""" print('Nina')

>>> my_name.__name__ 'wrapper'>>>> my_name.__doc__ # ... empty

@nnja

solution: use wraps, or wrapt library! !

from contextlib import wrapsdef say_adios(func): @wraps(func) # pass in which function to wrap def wrapper(): func() print('Adios!') return wrapper

@say_adiosdef say_max():""" Says the name Max""" print('Max')

>>> say_max.__name__ 'say_max'>>> say_max.__doc__ ' Says the name Max'

@nnja

Decorators: Common uses— logging— timing— validation— rate limiting— mocking/patching

@nnja

ContextDecorators ContextManagers+ Decorators combined.

@nnja

As of python 3.2 ContextDecorators are in the standard library. They're the best of both worlds!

— By using ContextDecorator you can easily write classes that can be used both as decorators with @ and context managers with the with statement.

— ContextDecorator is used by contextmanager(), so you get this functionality

✨

automatically

✨

.

— Alternatively, you can write a class that extends from ContextDecorator or uses ContextDecorator as a mixin, and implements __enter__, __exit__ and __call__

— If you use python2, a backport package is available here: contextlib2

@nnja

Remember @contextmanager from earlier?

from contextlib import contextmanager

@contextmanagerdef feature_flag(name, on=True): old_value = feature_flags.is_on(name) feature_flags.toggle(name, on) yield feature_flags.toggle(name, old_value)

@nnja

use it as a context manager

def get_homepage_url(): beta_flag_on = feature_flags.is_on(FeatureFlags.SHOW_BETA) return '/beta' if beta_flag_on else '/homepage'

with feature_flag(FeatureFlags.SHOW_BETA): assert get_homepage_url() == '/beta'

or use as a decorator

@feature_flag(FeatureFlags.SHOW_BETA, on=False)def get_profile_page(): beta_flag_on = feature_flags.is_on(FeatureFlags.SHOW_BETA) return 'beta.html' if beta_flag_on else 'profile.html'

assert get_profile_page() == 'profile.html'

@nnja

library I !

: freezegun lets your python tests ❇ travel through time! ❇

from freezegun import freeze_time

# use it as a Context Managerdef test(): with freeze_time("2012-01-14"): assert datetime.datetime.now() == datetime.datetime(2012, 1, 14) assert datetime.datetime.now() != datetime.datetime(2012, 1, 14)

# or a decorator@freeze_time("2012-01-14")def test(): assert datetime.datetime.now() == datetime.datetime(2012, 1, 14)

read the source sometime, it's mind-bending!

@nnja

NamedTupleUseful when you need lightweight representations of data.

Create tuple subclasses with named fields.

@nnja

Example

from collections import namedtuple

CacheInfo = namedtuple( "CacheInfo", ["hits", "misses", "max_size", "curr_size"])

@nnja

Giving NamedTuples default values

RoutingRule = namedtuple( 'RoutingRule', ['prefix', 'queue_name', 'wait_time'])

(1) By specifying defaults

RoutingRule.__new__.__defaults__ = (None, None, 20)

(2) or with _replace to customize a prototype instance

default_rule = RoutingRule(None, None, 20)user_rule = default_rule._replace(prefix='user', queue_name='user-queue')

@nnja

NamedTuples can be subclassed and extended

class Person(namedtuple('Person', ['first_name', 'last_name'])): """ Stores first and last name of a Person""" __slots__ = ()

def __str__(self): return '%s %s' % (self.first_name, self.last_name)

>>> me = Person('nina', 'zakharenko')

>>> str(me) 'nina zakharenko'

>>> me Person(first_name='nina', last_name='zakharenko')

@nnja

Tip

Use __slots__ = () in your NamedTuples!

— It prevents the creation of instance dictionaries.— It lowers memory consumption.— Allows for faster access

@nnja

"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take

away." — Antoine de Saint-Exupery

@nnja

New Tools— Magic Methods

— make your objects behave like builtins (numbers, list, dict, etc)

— Method ❇Magic❇— alias methods— * getattr— functool.partial

@nnja

— ContextManagers— Close resources

— Decorators— do something before/after call, modify return value

or validate arguments— ContextDecorators

— ContextManagers + Decorators combined!

@nnja

— Iterators & Generators— Loop over your objects— yield

— NamedTuple— Lightweight classes

@nnja

Don't be a mindless Code Monkey

@nnja

Use thesetools to bean elegantPythonista!

@nnja

Thanks!

@nnja

nina@nnja.io

bit.ly/elegant-python-ca

@nnja