17 June 2019© MARKLOGIC CORPORATION

Tim Murphy

Esri, Contextual Intelligence

James Kerr

Senior Technical Director, Partner Delivery,

MarkLogic

Tom Diepenbrock

Senior Principal Consultant, MarkLogic

Esri and MarkLogic: Building a Geospatial Data Hub

EsriGlobal Leader in GIS

• Annual Revenue > $1.4 Billion

• Privately held with zero debt

• 4200 U.S. employees – 6200 overseas

• 350,000+ Organization Customers

• 6.7 Million ArcGIS Online Users

• 75% of Fortune 500

• Supporting 130+ Countries

• 2500+ Partners

• Reinvests >31% in R&D

Apps

Desktop

APIs

ArcGISA Comprehensive Geospatial Platform

Supporting Individuals, Teams, and Organizations

Enterprise & Online

Dashboards

Story Maps

Social MediaViewer AppsAnalytic Apps

Field Apps

Across Organizations and Beyond

Desktop Pro App

Web Services-Based

Supports Apps That Extend the Power of Your Data to Everyone

Portal

ArcGIS Is Open and Interoperable

Certifications

. . . Successfully Integrated Into Thousands of Systems

AutoCAD

SAP HANA

Adobe Creative Cloud AWS

AzureMS Office

Oracle

SQL Server

Teradata

NetezzaSharePoint

Jupyter Notebook

Power BI

R Python

Revit PostgreSQL

Altibase

Dameng SQLite

Extensible ArchitectureOpen-Source Contributions

Open APIs & SDKsOpen Data Access Open-Source Integration

REST

OPeNDAPWMS

WCS CSW

GML

WFS

KMLWMTSSLD

GeoPackage

SOAP

IFC

LAS

netCDF

SQL

JSON WaterML CityGML

OneGeologyLERC

Shapefiles

Web Scene (I3S)

INSPIREXLSForm

WPSIMDF

Open Standards and Formats

Direct Product Integration

Open Software Architecture

Embeddable

. . . Many

Hadoop

developers.arcgis.com

Web Apps

App Builders

Smart

Mapping

Portals

Web Maps

Online

Content

Hub

Open Data

Solution Apps

Distributed

Story Maps Visualization

Field Apps

Location Analytics

. . . a Whole New Approach

. . . Fueled by Powerful Data and Analytics

Web GISEasy, Accessible, and Interconnected . . .

Connecting to Non-Traditional GIS Sources

+

Global Leader in GIS

Multi-model data integration

=

Location Analytics for ALL your Data

+

MarkLogic Esri Integration

MarkLogic Esri Connector

MARKLOGIC

ESRI

CONNECTOR

JSON

XML

RDF

TEXT

GEOSPATIAL

BINARY

ETC.

FEATURE

SERVICEFEATURE

SERVICEFEATURE

SERVICE

Optic API and TDE

§ Single, integrated platform – single, unified

query interface

§ Access any shape of data, or use all

together

§ Combine strengths of each underlying

query mechanism

SEARCH SQL SPARQL

LENSES (TEMPLATES)

OPTIC API

DOCUMENTS(JSON OR XML)

Data Architecture

Layer

FeatureFeature

VIEW

ASK ANYTHING

UNIVERSAL INDEX

DOCUMENTS (JSON/XML)Feature

Row

GeometryGeometry

GEOSPATIAL

INDEX

GeometryGeometry

ROW INDEX

FEATURE SERVICE

SPARQL RESULTS

FeatureRow

TRIPLE INDEX

Esri Connector Architecture

TEMPLATES

VIEWS

SPARQL

OPTIC API

DOCUMENTS (JSON/XML)

GEO QUERY SERVICE

Service DescriptorService

DescriptorService

Descriptor

LAYER METADATA SERVICES

G

E

O

D

A

T

A

S

E

R

V

I

C

E

S

KOOP

MARKLOGIC KOOP

PROVIDER FEATURE

SERVICE

GEOSPATIAL

ASK ANYTHING

UNIVERSAL INDEX

ARCGIS PRO ADD-IN

Service Descriptors

§ Define the Feature Services

§ Configure Layers

§ Set the Data Sources for each Layer

§ Set the Bounding Query for each layer

§ Set what Geospatial Indexes to use

§ Declarative way to generate Optic API

plans

Service Descriptor

{…

}

From SPARQL

From

View

JoinCTS

Where GeoJSON Feature

GeoJSON

Feature

Optic Query Pipeline

From

View

Optic Where

GeoJSON

Feature

Convert SQL WHEREto Optic expressions

Fetch geometry from the documents (if requested

and required)

Order

ByOffset Limit

Join

DocSelect

View 1

SPARQL

View n

From SPARQL

From

View

JoinCTS

Where

Optic Aggregation Pipeline

From

View

Optic Where

View 1

SPARQL

View n

Group

By

Convert Esri

group by and

stats parameters

to Optic

expressions

Order

ByOffset Limit GeoJSON

Feature

GeoJSON Feature

Stats JSON

§ Expose the power of full-text indexing, geospatial indexing and row-based

indexing via one interface

§ Configuration-based

§ Standard interface to expose data to Esri tools

§ Try it out

- MarkLogic Koop Provider: https://github.com/koopjs/koop-provider-marklogic

- ArcGIS Pro Add-in: Available on request

Summary

Demos

Real World Example

Observation Management System

Create, Manage, Search/Discovery, Analysis

Graph-based system

§ Complex, well-defined ontology

§ Approximately 600 object classes

§ Metadata, container, etc. classes

§ Mission-specific classes

§ Production Object Data

- Approximately 850M Triples

- 13M JSON-LD documents

Pain Point: Search

§ Poor graph DB performance/accuracy

MarkLogic Implements Search

§ Text search over JSON-LD

§ JSON-LD -> Triples

§ Geospatial, Temporal, Text Searches

§ Java Client API

Pain Point: Esri Integration

§ Esri + RDMS backend

§ Complex ETL

§ Brittle code, high maintenance

§ RDMS forced implementation details in

graph data

§ Users never got what they wanted

Graph DB à Relational DB ETL Process

MarkLogic Esri Connector Solution

Observation

MarkLogic SPARQL

Document Search

Observation

Observation

Observation

MarkLogic Esri

Connector Services

Multi-Model Database

§ All the data is already in MarkLogic

§ Added TDE

§ Added MarkLogic Esri Connector

§ No ETL

Ontology Driven Templates

§ Dynamically generated TDE Templates

§ One template per object class

§ ≈600 object class templates

§ ≈300 mission-specific class templates

§ Templates update with ontology changes

§ One denormalized view, ≈700 sparse

columns defined

§ ≈150 columns actually populated with data

VIEW

Ontology Driven Service Descriptors

§ Dynamically generated Service Descriptors

§ ≈600 static services based on object

classes

§ ≈100 Services based on specific mission

classes

§ 3 – 54 Layers per Service Descriptor

§ 30 attributes per layer

§ Columns with data identified via Optic API

aggregate functions

§ Service descriptors update with ontology

changes

Service DescriptorService

DescriptorService

Descriptor

Benefits

No Baked in Silos Multi-Model Flexibility Core MarkLogic Capabilities

HA ACID Scalable

Search Secure Triples

Geo TDE Cloud

Benefit:

Painless Esri Integration

Service On The Fly

Unforeseen Benefit

§ “Make me a personalized service for any observation created in the last ten

days mentioning the name ‘Humpty Dumpty’ within 30 words of the the word

‘horses’

§ “Show me all objects of type ‘Vehicle’ created in the last month that are

garaged at buildings of type ‘Single Family Home’ with four bedrooms”. **The

fact that the building is a Single Family Home may be in a different document

than the fact that the building has four bedrooms**

Service On The FlySearch Examples

End Users Refine Searches

Client App ServerTransactional Apps

Application Inserts Service Descriptors

Client App ServerTransactional Apps

Application Inserts Service Descriptors

Client App ServerTransactional Apps

Completely new capabilities___________________________________________________________________________________________________________________________________________________________________________

ü “Breaks the bonds” of static services

ü Flexible Feature Attribution

• Based on what matches the backing query

• No longer tied to object class definitions

ü User-Specific Services

ü Service Discovery

ü Secure

• Retrieve only features each user is allowed to see at time of access

• Discover only services each user is allowed to see

___________________________________________________________________________________________________________________________________________________________________________

Users finally get what they want

What’s Next?

§ MarkLogic added to ICAP

§ ArcGIS Pro add-in published to the Esri Marketplace

§ Connector enhancements

- Support for time-aware layers

- SQL “bounding query”

- Improved deployment

Next Steps

Questions?

Thank you