Making the entire world accessible via a single API

Ed FreyfogleBarcelona Data Engineering - 30 Nov. 2017

http://opencagedata.com@opencagedata

- Some background

- Technical overview

- Build versus Buy?

- Questions

Overview

$whoami

$whoami- Co-founder of OpenCage

- Before that, co-founded a real estate metasearch engine called Nestoria. 10 years, 20 employees, €4.5M in revenue

- Way, way before that, first developer at yahoo.de

- Moved to BCN to live the good life

opencagedata.com

WTF is geocoding?

Where are we?

Barcelona

BarcelonaIronhack BCN

BarcelonaIronHack BCN

08018

BarcelonaIronHack BCN

08002Carrer de Pamplona

Poblenou

BarcelonaIronHack BCN

08002Carrer de Pamplona

Poblenou

Europe

BarcelonaIronHack BCN

08002Carrer de Pamplona

Poblenou

Europe

Spain / Rep. of CataloniaNEW!

41.39778,2.19011

placename lat/long

placename long/lat

forward

placename long/lat

forward

reverse

Why Geocode?- Vehicle tracking

- Routing

- Data enrichment

- Put points on a map

- Image processing

Yes!

Yes! but …T&Csexpensive at volume

OpenStreetMap

Yes!

Yes! but …not for high volume usevariable coverage

Yes!

Yes! and …one API, many geocodersfree as in freedomenterprise level reliability

- Some background

- Technical overview

- Build versus Buy?

- Questions

Overview

OpenStreetMapwww.openstreetmap.org

OpenStreetMap- Crowdsourced geodata. Anyone can edit. 4 million reg. users

- Bad name. Is a database, not a map.

- 3 million edits per day. And growing.

- Operations run by volunteers.

- Master DB is hosted at a university in London

- Data is under ODbL License

- Many, many different tools for editing, querying, visualizing the data

Example

$ curl ‘https://api.opencagedata.com/geocode/v1/json?q=41.39778,2.19011&key=...&pretty=1’

Example

$ curl ‘https://api.opencagedata.com/geocode/v1/json?q=41.39778,2.19011&key=...&pretty=1’

"components": { "ISO_3166-1_alpha-2": "ES", "_type": "pub", "city": "Barcelona", "city_district": "Sant Martí", "country": "Spain", "country_code": "es", "county": "BCN", "house_number": "96", "postcode": "08018", "pub": "Sonora Sport Tavern", "road": "Carrer de Pamplona", "state": "Catalonia", "suburb": "el Parc i la Llacuna del Poblenou" }, "formatted": "Sonora Sport Tavern, Carrer de Pamplona, 96, 08018 Barcelona, Spain",

https://geocoder.opencagedata.com/code

Libraries for - Perl, PHP, Python, Ruby, R, Java, .NET, elixir,

Haskell, Go, Erlang, nodejs, Stata, perl6, iOS

- Leaflet, OpenLayers

TMTOWTDI

ArchitectureHTTP request

ArchitectureHTTP request LB

LB

- HAProxy- Two different physical locations

ArchitectureHTTP request LB

LBHTTP request LB

meta

- ubuntu- apache- mod_perl

ArchitectureHTTP request LB

LBHTTP request LB

meta

Kong

HTTP request

- authentication- validity checking- routing

ArchitectureHTTP request LB

LBHTTP request LB

meta

Kong

HTTP request

geocoder geocoder geocoder

ArchitectureHTTP request LB

LBHTTP request LB

meta

Kong

HTTP request

geocoder geocoder geocoder

OSMContinually updating

HTTP request

ArchitectureHTTP request LB

LBHTTP request LB

meta

Kong

HTTP request

geocoder geocoder geocoder

OSMContinually updating

- accept response, error check- standardize format- rank- add annotations- send to client in XML, JSON, etc

- timezones - geohash, MGRS, Maidenhead, DMS- what3words- calling codes, currency- and more ...

Annotations

Formatting

$ curl ‘https://api.opencagedata.com/geocode/v1/json?q=41.39778,2.19011&key=...&pretty=1’

"components": { "ISO_3166-1_alpha-2": "ES", "_type": "pub", "city": "Barcelona", "city_district": "Sant Martí", "country": "Spain", "country_code": "es", "county": "BCN", "house_number": "96", "postcode": "08018", "pub": "Sonora Sport Tavern", "road": "Carrer de Pamplona", "state": "Catalonia", "suburb": "el Parc i la Llacuna del Poblenou" }, "formatted": "Sonora Sport Tavern, Carrer de Pamplona, 96, 08018 Barcelona, Spain",

115 New Cavendish Street, London W1T 5DU, United Kingdom

223 William Street, Melbourne VIC 3000, Australia

Rosenthaler Straße 1, 10119 Berlin, Germany

Via Pisacane, 13, 76121 Barletta BT, Italy

3 Upper Alma Road, Rosebank, Cape Town, 7700, South Africa

Formatting

ArchitectureHTTP request LB

LBHTTP request LB

meta

Kong

HTTP request

geocoder geocoder geocoder

OSMContinually updating

- logging- monitoring- emails

metrics

scp

Monitoring

We use StatusCake

- Dev/testing process

- Deployment

- User registration / onboarding

- Billing

Not covered

- Some background

- Technical overview

- Build versus Buy?

- Questions

Overview

- Unable to buy- Lack of finance- Lack of affordable, reliable vendor

When to build

- Unable to buy

- Core to your service

When to build

- Unable to buy

- Core to your service- Critical differentiator- Privacy / data protection issues- Legal requirement

When to build

- Unable to build

When to buy

- Unable to build

- Need to move fast

When to buy

- Unable to build

- Need to move fast

- Is non-core

When to buy

- Unable to build

- Need to move fast

- Is non-core

- Unable to maintain

When to buy

- Financial cost

- Opportunity costs

- Staffing costs

Cost analysis

- Industry trend has been towards buying- Servers (AWS, Dropbox, etc)- Login / Identity (FB, google, etc)- Billing (Stripe)- 1000s of SaaS businesses

The trend

- Engineers love to build

MAIN POINT

- Engineers love to build

- Building is (comparatively) easy

MAIN POINT

- Engineers love to build

- Building is (comparatively) easy

- MAINTENANCE IS REALLY HARD!!!!

MAIN POINT

- Some background

- Technical overview

- Build versus Buy?

- Questions

Overview

Three more things

Three more things1. Lifetime 10% discount

use code: XXXXXX

Three more things1. Lifetime 10% discount

use code: XXXXXX

2. Happy to pay anyone who wants to writea wrapper library: F#, Scala, Lua, etc

Three more things My friends at AVUXI are hiring http://www.avuxi.com

Ed FreyfogleBarcelona Data Engineering - 30 Nov. 2017

https://opencagedata.com@opencagedata

Thanks! Questions?