OS MasterMap it’s not a map – but data

Ian Painter

Snowflake Software

About me

• I don’t represent Ordnance Survey, • I worked there for 10 years• My opinions are my own

First Some History

• County Series was Ordnance Survey’s large scale paper product

• Landline was Ordnance Survey’s first digital product

• Built to print paper maps quicker• Blind digitised … so plenty of

And then came OS MasterMap

Miracle Occurs

Fastrack

• Take Landline as input• Clean it up … like really clean it up • Stitch it together (edgematching)• Polygonise it• Restructure the road network• Beef up the attribution• Multi-level structures• Classify all the polygons• Associate all the cartographic text• Only 1% manual editing• Complete in a year

Geospatial Object Server

All very well creating all this stuff but we need somewhere to put it:

•Store it all seamlessly•Maintain the topology•Store change•Seamless ordering•Huge data volumes•From 100 features to 450 million•Built on an object oriented database called ObjectStore

Fastrack + GOS =

• Unique identifiers• Real world• Seamless Product• Change-only-update• Delivered as GML

Impacts on the Industry

• Data management– From files to databases

• Large data volumes• Complex data models

– From simple features to table joins, multiple geometries

• GML– XML rather than proprietary

• Change only update– Individual feature update rather than file replacements

Key Market Selling Points

• A data product– Clean– Structured– Rich attribution– Unique identifiers

• Seamless - designed for query and analysis• Change Intelligence

– Change triggers– Historical archives

How did we do 10 years on

• Map vs data– Coloured backdrop map– Cloud web mapping is step backwards

• Change Only– Slow start but most now applying COU– Little use of COU for change intelligence queries

• Identifiers– Core referencing hasn’t really worked

• Seamless– Very little spatial analysis

But Why?

• Data model capabilities of GIS are very limiting• Too much focus on web mapping

– Even more so with online mapping portals

• Proprietary nature of GIS– The limitation of it’s formats– Preference for file based data management– Lack of integration with mainstream IT

• Functionality is focused on the map, not the data• Huge hardware requirements for spatial analytics

So what’s going to change all this?

When are we going to drop the map?

Isn’t it just data? Big Data?

Well it’s not Big Data … but I needed a link!

• Big Data is a buzz word!• An technology paradigm to massage your ego• Everybody likes to have something … BIG

Seriously … a Big Data 101

• Big Data looks at data in two ways1. Structured Data – think schema, data models

2. Unstructured Data – free text, insurance claim, transaction log

• Structured Data tends to be stored in database– But not any old database … a NOSQL database

• Unstructured Data tends stored on a file system– But not any old file system … a distributed filesystem– And then processed through a paradigm called MapReduce

So what’s all the tech

•NOSQL Databases:– Columnar, Document, Key-Value and Graph – Netezza, Vertica, Terradata, MongoDB,

•Unstructured data:– Hadoop: hdfs, MapReduce, Amazon Elastic MapReduce

•Can also be hardware– Lot’s of hardware

Who uses this stuff

• Yahoo created Hadoop, it runs: – Facebook, Twitter and eBay– Heavy use in Telco, Finance and retail

• Facebook, runs the largest Hadoop cluster in existence– 21PB of storage, – 2000 (8 core) machine, – 12TB per machine, – 32Gb of RAM

But what about Geo support?

• IBM Netezza has native spatial• MongoDB• ESRI ArcGIS 10.1 has native Netezza support

• But that’s only required if we think of geo as map• Geo won’t be just maps for much longer …• Geo is just collection of relationships

– This is next to this, this is inside that

• NOSQL database are far better suited to graphs

A Big Data problem …

• NFC – Near Field Communication will put a spatial component of every cash transaction

• Think of running a spatial query against national OSMM to check if the NFC transaction happened in the correct location – So that’s 1 x,y against 440 millions features

• NFC has the potential to overtake cash in less than 10 years– We’re talking 100’s millions vs 100’s of millions

a minute … now that’s a Big Data problem

GIS ain’t gonna cut it

• Step up – Big Data– Geo as graphs and relationships

• Take a seat– GIS and rest your tired data model capabilities– Maps and cartography– Relational databases

• Use Geo to give me an answer but don’t show me a map

In summary

• Despite OS MasterMap being over 10 years old – It’s still ahead of GIS capabilities– A lot of inherent value still isn’t used

• Spatial is not special it’s just another data type• Too much focus on a map to solve all• Big Data is coming, Geo will be very different• Some sectors will skip GIS altogether• Think data, not maps• Graphs, not geometry

Ian Painter

Snowflake Softwareian.painter@snowflakesoftware.com

http://www.snowflakesoftware.com

@iapainter

Ya scallway whut deserves the black spot. Come t’ our stand an’ be seein’ some great demos.