Date post: | 16-Jul-2015 |
Category: |
Technology |
Upload: | wellfire-interactive |
View: | 52 times |
Download: | 0 times |
Crafting [Better] API Clients
PyTennesee 2015- Ben Lopatin
Partner and developer @ Wellfire Interactive
@bennylope
Clients, not Services
Why?
A great API is necessary but insufficient to be
useful.
This is the right way.
This is the right way. This is some good ideas.
Acknowledge that you’re making design decisions.
Some assumptions
Data Model
Why?
Database API
Real world
Layers of abstraction!
Connections & data
Designing the Python data interface
Explicit data attributes or implicit collection?
class DataResource: def __init__(self, id, first_name, last_name): self.id = id self.first_name = first_name self.last_name = last_name resource.first_name
resource = { 'id': 1, 'first_name': 'Ben', 'last_name': 'Lopatin', } resource['first_name']
class DataResource(dict): def update(self, client): # Do something cool here client.update(self.id, self.first_name, self.last_name)
Data or Resources?
• If you think about everything as a resource, then shouldn’t it have the same methods?
• Or is it data and the API is just providing some
• You’re not obliged to represent 1:1
Be mindful of what you discard
Second, we wanted to give you a heads up that we're announcing a new feature soon -- additional fields. This will allow people to get congressional districts, state legislative districts, timezones, and school districts with a forward or reverse lookup. We are looking to add more additional fields in the future (namely Census data).
Errors
Why?
What?
API application errors Authentication errors
Account errors …
If only there were a concept to encapsulate
errors in Python…
…that would be exceptional!
Exceptions for obvious flow control
try: response = client.request()except APIPaymentError: # Email corporate accounts payable raiseexcept APIAuthError: # Redirect to account settings raise
Map exceptions to API errors
class SmartyStreetsError(Exception): """Unknown SmartyStreets error""" def __str__(self): return self.__doc__class SmartyStreetsInputError(SmartyStreetsError): """HTTP 400 Bad input. Required fields missing from input or are malformed."""class SmartyStreetsAuthError(SmartyStreetsError): """HTTP 401 Unauthorized. Authentication failure; invalid credentials"""class SmartyStreetsPaymentError(SmartyStreetsError): """HTTP 402 Payment required. No active subscription found."""class SmartyStreetsServerError(SmartyStreetsError): """HTTP 500 Internal server error. General service failure; retry request."""
ERROR_CODES = { 400: SmartyStreetsInputError, 401: SmartyStreetsAuthError, 402: SmartyStreetsPaymentError, 500: SmartyStreetsServerError,}
…maybe not all errors
Errors aren’t necessarily limited to HTTP status
codes
Exceptions +
Error codes
Where?
Logging
Why?
What?
Verify requests issued
Request/response count
API performance
API errors
How?
@decorators?
@log_requestsdef _req(self, method='get', verb=None, headers={}, params={}, data={}): url = self.BASE_URL.format(verb=verb) request_headers = {'content-type': 'application/json'} request_params = {'api_key': self.API_KEY} request_headers.update(headers) request_params.update(params) return getattr(requests, method)(url, params=request_params, headers=request_headers, data=data)
def log_requests(func): def decorator(*args, **kwargs): request_id = str(uuid.uuid4()) logger.info("Requesting %s" % request_id) try: resp = func(*args, **kwargs) except: logger.exception("Request error %s" % request_id) raise logger.info("Response %s" % request_id) return resp return decorator
Testing
Why?
How?
Live API Mock servers
Mocked responses
That’s not nice (or fun)
Mock server?
Live testing’s hard, let’s go mocking!
Mocking requests
responses HTTPretty
responses HTTPretty betamax
@responses.activate def test_auth_error(self): responses.add(responses.POST, 'https://api.smartystreets.com/street-address', body='', status=401, content_type='application/json') self.assertRaises(SmartyStreetsAuthError, self.client.street_addresses, [{}, {}])
@responses.activate def test_payment_error(self): responses.add(responses.POST, 'https://api.smartystreets.com/street-address', body='', status=402, content_type='application/json') self.assertRaises(SmartyStreetsPaymentError, self.client.street_addresses, [{}, {}])
@httpretty.activate def test_auth_error(self): """Ensure an HTTP 403 code raises GeocodioAuthError""" httpretty.register_uri(httpretty.GET, “http://api.geocod.io/v1/parse", body="This does not matter", status=403) self.assertRaises(GeocodioAuthError, self.client.parse, "")
There can be benefits to some live testing
Performance
Connections
Batching
The right API methods
Data structures
class APIData: def init(self, id, name, image): self.id = id self.name = name self.image = image
class APIData: __slots__ = ['id', 'name', 'image'] def init(self, id, name, image): self.id = id self.name = name self.image = image
Concurrency
response_iter = ( grequests.post( url=url, data=json.dumps(data_chunk), params=params, headeresponse_iter=headeresponse_iter, ) for data_chunk in chunker(data, 100) ) responses = grequests.imap(response_iter, size=parallelism)status_codes = {}addresses = AddressCollection([])for response in responses: if response.status_code not in status_codes.keys(): status_codes[response.status_code] = 1 else: status_codes[response.status_code] += 1 if response.status_code == 200: addresses[0:0] = AddressCollection(response.json())
session = FuturesSession(max_workers=parallelism)session.headers = headerssession.params = paramsfutures = [ session.post(url, data=json.dumps(data_chunk)) for data_chunk in chunker(data, 100) ] while not all([f.done() for f in futures]): continuestatus_codes = {}responses = [f.result() for f in futures]addresses = AddressCollection([])for response in responses: if response.status_code not in status_codes.keys(): status_codes[response.status_code] = 1 else: status_codes[response.status_code] += 1 if response.status_code == 200: addresses[0:0] = AddressCollection(response.json())
You know Twisted can do that, right?
Security
HTTPS
Python 2.7
• pyOpenSSL
• ndg-httpsclient
• pyasn1
Authenticating
Docs
Django
No?
Python library
Anything Django related
• Create the client *first* then Django integration is a bonus.
• Mapping models to a distant API.
• It’ll be easier for you to maintain and test
• Easier for other people to use for other things
class SomeModel(models.Model): my_field = models.CharField(max_length=100) def sync(self, client=None): if client is None: client = APIClient() client.update(id=self.id, name=self.my_field)
Code Gen
Starting from (DSL) API documentation, automatically
generate matching client code
Crank ‘em out
In [hypothetical] practice?
Suitability?
The End