24
Mappy- ing Open Source 09/21/2013, FOSS4G2013
| Date post: | 17-Dec-2014 |
| Category: |
Technology |
| Upload: | audrey-malherbe |
| View: | 2,189 times |
| Download: | 0 times |
Mappy-ing Open Source 09/21/2013, FOSS4G2013
367 millions route requests in 2012
4 millions referenced business
Mappy, a brand of SoLocal Group
About 10 millions monthly users
27 millions monthly visits
Provide route planner and local services on the web and
mobiles devices
Feedback on the overhaul of the mapping engine of Mappy around Mapnik and PostGIS
Plan
The path
Our journey
Feedback
– Les enjeux du projet
The path
The path – starting point
Background
– Map processing for 16 years with TomTom/AND databases
– Maps are used by Mappy & French YellowPages media
– « home-made » processing mastered but ageing
The future
– Repositionnning map to fit new requirements : the purpose of the
map is now to support content
– Open-source-oriented strategy (moving towards)
– Anticipate emerging challenges (vector tiles, HD maps, relief, ...)
Geographic data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
Data processing
Mostly pre-computation
Duration : 10 days
Non standard
proprietary format
12 zoom levels
Software system
Fully mastered (behaviour)
Some dead code
Costly evolutions Non negociable performances
6 Billions req/month
BACK OFFICE PRODUCTION
The path - goals
– Create a map design able to support content
– Be ready to release with the new version of our website
fr.mappy.com
– Reduce the computational time
– address weaknesses of the platform and keep in mind its strengths
– Streamlining map tile production
– Standardizing (REST API, etc.)
– Build for the future (Able to scale and support new services)
Take a Deep Breath. Start Again.
Geographic
data
API AJAX
NewMappy
SDK Mobile
New Mappy KyotoCabinet KyotoCabinet
New platform
in production
Reduced Preparation Time
2days (vector) + 2days (relief)
Standard
Live code
Mapnik contribution
Reduced operating cost
BACK OFFICE PRODUCTION
– Le chemin
Our journey
Our journey - Starting point
– Heterogeneous knowledge of the team
on open source technologies
– Training & getting help from expert
– Bring a culture of open source
– Split project
– Migrate brick by brick in order to learn and master the platform
– « Eat our own food »
– Test it fast
Innophi
Geographic
data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
API AJAX
NewMappy
SDK Mobile
New Mappy KyotoCabinet KyotoCabinet
BACK OFFICE PRODUCTION
BACK OFFICE PRODUCTION
Templates
Step 0
Map design
By marketing and Map designer
Geographic
data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
API AJAX
NewMappy
SDK Mobile
New Mappy KyotoCabinet KyotoCabinet
BACK OFFICE PRODUCTION
BACK OFFICE PRODUCTION
Step 1
Migration of database and map engine
Keep Front interface
Work on database performance
Geographic
data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
API AJAX
NewMappy
SDK Mobile
New Mappy KyotoCabinet KyotoCabinet
BACK OFFICE PRODUCTION
BACK OFFICE PRODUCTION
Step 2
Cache migration
New APIs
Add relief /HD tiles
Work on performance
and configuration (Cache/Service/APIs)
Geographic
data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
API AJAX
NewMappy
SDK Mobile
New Mappy KyotoCabinet KyotoCabinet
BACK OFFICE PRODUCTION
BACK OFFICE PRODUCTION
Step 3
Streamlining databases
Work on scaling
Optimizing process
Our journey – key points
– Be able to challenge historical choices
– Reach a better consensus with our customers (Web/Mobile)
– Less time on tools implementation than configuration
– We built prototype in two weeks
– We worked on configuration during 4 months
– Real issue : association of metadata
with map tiles
(dynamic display on map icons)
Our journey – technical choices
– PostGIS: transactional database/scalable/standard compliant
– Mapnik : approved by marketing and technical teams!
– Tornado :
– High performance reputation, already used by mobile team
– Varnish :
– Highyield
– Already used
by front teams
– Nothing to
prove
Our journey – Contribution to Mapnik
– Mappy was legitimate to participate in an open source project
– Develop an open source policy at Mappy
– Arrange technical organization (Open Source Manager, Github
website : https://github.com/Mappy)
– Contribute to Mapnik : implementation of parallelism with PostGIS
(available on master). Performance gain: 25%
– Some tools availables on Github like pycnik
Feedback
Feedback– Key figures
– Production platform handles 85% of the load (All media except
Mobile YP)
– 14 servers:
– Varnish : 2 / Tornik : 8 / PostGIS : 4
– Output without cache:
– Peak 240 req/s
– Average throughput 150 req/s
– Output with cache: 2500 req/s (peak)
– Cache Hit Ratios: 94%
– Cache preload: 2 millions objects
– Postgis database: 75 Go
– Relief database: 300 Go
Feedback… team
– Positive
– Team motivation, proud and in control of our new platform
– PostGIS and Mapnik tools
– Contribution/open source world
– Negative
– Difficult to identify correct projects splitting
– Contribute lately to Mapnik
– Contributing requires available time (discussion with
community/ pull requests)
– Deployment by our customer not as fast as expected
Thank you
@AudreyMalherbe https://github.com/Mappy
![SBC - Proceedings of SBGames 2013 Computing Track Ð Short ...€¦ · Factory [5] and Mappy [6]. Newer alternatives include Tile Studio [7] and Tiled [8] (both are free software).](https://static.documents.pub/doc/80x56/5fe2c146ebe335656c2e5015/sbc-proceedings-of-sbgames-2013-computing-track-short-factory-5-and-mappy.jpg)
![SBC - Proceedings of SBGames 2013 Computing Track Ð Short … · 2013. 10. 25. · Factory [5] and Mappy [6]. Newer alternatives include Tile Studio [7] and Tiled [8] (both are free](https://static.documents.pub/doc/80x56/5fe2c1eb344f8b3ad8410c01/sbc-proceedings-of-sbgames-2013-computing-track-short-2013-10-25-factory.jpg)
