GeoEvent & GeoAnalytics - Esri€¦ · -the geohash spatial index only supports GCS WGS 1984,...

GeoEvent & GeoAnalytics:Leveraging the spatiotemporal big data store

Adam MollenkopfReal-Time & Big Data GIS Capability Lead, Esri

[email protected]

@amollenkopf

Suzanne FossReal-Time & Big Data Product Engineer, Esri

[email protected]

@sfoss_esri

mailto:[email protected]

1

2

3

4

5

6

Agenda

why a new type of data store?

visualizing observation data

writing observation data from GeoEvent Server

on-the-fly aggregations

writing analytic results from GeoAnalytics Server

deeper dive

why a new type of data store?1

Web GISwith real-time capabilities

Web GIS

ingest visualize

analyze store

real-time & big data

thousands3K e/s

3K e/s 200 e/s

BYO-RDBMS

X

X

200 e/s

GeoEvent

Server

Web GISwith real-time capabilities

Web GIS

ingest visualize

analyze store


GeoEvent

Server

thousands3K e/s

3K e/s 200 e/s

relational

data store

X

X

200 e/s

Web GISwith real-time & big data capabilities

Web GIS

spatiotemporal

big data store

ingest visualize

analyze store


GeoEvent

Server

millions4K e/s

4K e/s 10Ks e/s

4K e/s

Web GISwith real-time & big data capabilities

Web GIS

spatiotemporal

big data store

ingest visualize

analyze store


millions4K e/s

4K e/s 10Ks e/s

GeoEvent

Server

4K e/s

4K e/s

4K e/s

ArcGIS Enterprisewith real-time & big data capabilities

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big DataIoT

ingest visualize

analyze store


millions4K e/s

4K e/s 10Ks e/s

GeoAnalytics

Server


ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big DataIoT

GeoAnalytics

Server

1

2 43

MINIMUM environment

functional servers & spatiotemporal big data store

SHOULD BE on ISOLATED machines!!!


ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big DataIoT

GeoAnalytics

Server

1

2 3 64 75 8

RECOMMENDED environment

functional servers & spatiotemporal big data store

SHOULD BE on ISOLATED machines!!!


DesktopWeb Device

visualization

live & historic

aggregates & features

map & feature service

• Ingest high velocity real-time

data into ArcGIS

• Perform continuous analytics

on events as they are received

• Store observations in a

spatiotemporal big data store

• Run batch analytics on

stored observations

• Visualize high velocity &

volume data:

- as an aggregation

- as discrete features

• Notify those who need to

know about patterns of

interest

stream service

live features

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

storage

ingestion

analytics

GeoAnalytics

Server

analytics

visualizing observation data2

Observation datatypes of observation data

movingsomething

that moves

• planes

• vehicles

• animals

• satellites

• storms

stationarystands still but

attributes change

• water gauges

• weather stations

• traffic sensors

• air quality

discretesomething that

“just happens”

• crimes

• lightning

• accidents

Observation datadefined

• An observation is a recording of a feature’s attribute values and location at a specific

moment in time.

- Observations are immutable, they happen and are typically not edited.

- Observations can be replayed over space & time.

- Moving observations are identifiable by a unique attribute, known as a TRACK_ID.

space & time

AM

Observations can be

shown at a specific time

PM

Moving observations can

be identified by a unique attribute

tracks

1

1

1

2

22

3

3

3

the tracks of moving observations

Moving observation illustration for two tracks

over space (X, Y) and time (T)

can be reconstructed

T

On-the-fly aggregationsof observations

• The spatiotemporal big data store enables you to aggregate data on-the-fly

- aggregating on-the-fly accommodates for real-time data

- you can perform exploratory analysis via any combination of space, time and attribute queries

- toggle from aggregation to raw feature view

- access feature level attributes in aggregate or feature level view

Demo


visualizing observation data

writing observation data

from GeoEvent Server3

Writing observation datafrom GeoEvent Server

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

IoT


data source

input output

GeoEvent Service

Demo

into a spatiotemporal big data store

from GeoEvent Server

Writing observation data

on-the-fly aggregations4

Preparing data for on-the-fly aggregationsspatial indexing: geohash

• as data is written to a data source in the spatiotemporal big data store:

- a spatial index for geohash aggregation is continuously updated

geohash aggregation (based on a geohash index)

Preparing data for on-the-fly aggregationsspatial indexing: geohash & square


- a spatial index for geohash aggregation is continuously updated

- a spatial index for square aggregation is continuously updated

square aggregation (based on a square index)geohash aggregation (based on a geohash index)

Preparing data for on-the-fly aggregationsspatial indexing: triangle


- a spatial index for ‘pointy’ triangle aggregation is continuously updated

- a spatial index for ‘flat’ triangle aggregation is continuously updated

flat triangle aggregation (based on a flat triangle index)pointy triangle aggregation (based on a pointy triangle index)

Preparing data for on-the-fly aggregationsspatial indexing: hexagon (same as triangle)


- a spatial index for ‘pointy’ hexagon (pointy triangle) aggregation is continuously updated

- a spatial index for ‘flat’ hexagon (flat triangle) aggregation is continuously updated

flat hexagon aggregation (based on a flat triangle index)pointy hexagon aggregation (based on a pointy triangle index)

Preparing data for on-the-fly aggregationsspatial indexing


- up to four types of spatial indices are supported: geohash, square, pointy & flat hexagon/triangle

- this is in addition to a temporal index on the time field

- and an inverted index on each of the attribute fields

geohash

square

trianglepointy

triangleflat

hexagonflat

hexagonpointy

Visualizing on-the-fly aggregationsgeohash

geohash aggregation response geohash aggregation response

• map & feature services can query a data source in the spatiotemporal big data store with results

aggregated on-the-fly.

map servicefeature service

lodType=geohash&lod=2

feature servicelodType=geohash&lod=2&returnGeometry=false

geohash aggregation response

Visualizing on-the-fly aggregationssquare

map servicefeature service

lodType=square&lod=2

square aggregation response square aggregation response



Visualizing on-the-fly aggregationshexagon aggregation

map service

flat hexagon aggregation response

feature servicelodType=flatHexagon&lod=2

flat hexagon aggregation response



Visualizing on-the-fly aggregationsspatial projection

• some of the spatial indices support projection(s):

- the geohash spatial index only supports GCS WGS 1984, however you can project on-the-fly

- square, pointy & flat hexagon/triangle spatial indices are defined with a spatial reference

geohash aggregation in world mollweide projection (wkid = 54009) square aggregation in world mollweide projection (wkid = 54009)

Demo

geohash, square & hexagon

spatial projection

Visualizing on-the-fly aggregations

http://github.com/esri/aggregation-viewer-server-map-service http://github.com/esri/aggregation-viewer-client-feature-layer

writing analytic results

from GeoAnalytics Server5

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big Data

GeoAnalytics

Server

Writing analytic resultsfrom GeoAnalytics Server

IoT

GeoAnalytics Serveranalytic capabilities

Summarize DataAggregate PointsSummarize NearbySummarize WithinReconstruct TracksCreate Panel

Find LocationsFind Existing LocationsFind Similar Locations

Analyze PatternsCalculate DensityFind Hot SpotsCreate Space Time Cube

Use ProximityCreate Buffers

Manage DataCopy to Data StoreJoin Features

* time aware tools

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

IoT

GeoAnalytics

Server


• GeoAnalytics Server has the ability to:

- perform analytics against data sources in the spatiotemporal big data store

- and write the analytic results to the spatiotemporal big data store

Web GIS layers

new Web GIS layers

use your Web GIS layers through Pro, Portal,Python notebooks or the REST API

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

GeoAnalytics

Server

big datafile shares

featureservice

.shp

shapefiles

.csv

text files

Hadoopfile system

Hive


• GeoAnalytics Server has the ability to:

- perform analytics against files in an external big data file share

- and write the analytic results to the spatiotemporal big data store

new Web GIS layers

Web GIS layers

Loading datavia ‘Copy to Data Store’ GeoAnalytics tool

GeoAnalytics tool interface

output features

big datafile shares

.shp

shapefiles

.csv

text files

Hadoopfile system

Hive

featureservice

relationaldata store

spatiotemporalbig data store

Writing analytic resultsvia ‘Reconstruct Tracks’ GeoAnalytics tool

GeoAnalytics tool interface

big datafile shares

.shp

shapefiles

.csv

text files

Hadoopfile system

Hive

featureservice

relationaldata store

spatiotemporalbig data store

Writing analytic resultsvia ‘Reconstruct Tracks’ GeoAnalytics tool

output features

input features GeoAnalytics tool interface

deeper dive6- operating system requirements

- resource requirements

- data reliability

- ObjectId

Operating system requirements64-bit Windows or Linux

resource minimum RECOMMENDED notes

64-bit

operating system

(Windows)

Windows Server:

2012 R2 Standard & Datacenter (Sep 2016)

2012 Standard & Datacenter (Sep 2016)

2008 R2 Standard, Enterprise & Datacenter (sp1)

2008 Standard, Enterprise & Datacenter (sp2)

Windows Server:

2012 R2 Standard & Datacenter (Sep 2016)

‘_’ characters are

not allowed in

machine names

64-bit

operating system

(Linux)

Red Hat Enterprise Linux Server 7 (update 8)

Red Hat Enterprise Linux Server 6 (update 2)

CentOS Linux 7 (7.2)

CentOS Linux 6 (6.8)

Scientific Linux 7 (7.2)

Scientific Linux 6 (6.8)

Ubuntu Server LTS (16.04.1)

SUSE Linux Enterprise Server 12 (sp1)

SUSE Linux Enterprise Server 11 (sp4)

Oracle Linux 7 (update 2)

Oracle Linux 6 (update 8)

Use your Linux flavor of choice with the

latest update version Esri has certified.

http://server.arcgis.com/en/data-store/latest/install/windows/arcgis-data-store-system-requirements.htm

http://server.arcgis.com/en/data-store/latest/install/linux/arcgis-data-store-system-requirements.htm

Resource requirementsprocessors




processors 8 cores 16 cores

More processors results in

faster writes (indexing time),

higher write throughput &

faster query response time.

Resource requirementsprocessors, memory









storage 400 GB 1 TB or moreStorage need is dictated by data velocity & size.

SSD results in much faster writes & queries.

Resource requirementsprocessors, memory, storage











memory 16 GB 32 GBMore memory results in

higher likelihood of query cache hits.

32 GB is the maximum due to JVM limitations.

Resource requirementsprocessors, memory, storage & instances









memory 16 GB 32 GBMore memory results in

higher likelihood of query cache hits.

32 GB is the maximum due to JVM limitations

# of

ISOLATED

instances

1 3 or more

With one instance there is risk of data loss.

With three instances replicas are written

increasing data reliability.

The # of instances will vary based on the velocity of

incoming data as well as the data retention policy

configured on each data source.



spatiotemporal

big data store

GeoEvent

Server

Data reliabilityshards & replication factor

GeoAnalytics

Server

node 1

node 2

node 3node 4

node 5

T1

T3

T2

T2

r = 1

T1

T3

spatiotemporal

big data store

Data reliabilityauto-rebalancing of data upon node membership changes, + or -, in the big data store

GeoEvent

Server

GeoAnalytics

Server

node 2

node 3node 4

node 5

T1

T3

T2

T2

r = 1

x

T1

T3

T1

T1T3

purge based on

data retention

spatiotemporal

big data store

GeoEvent

Server

Data reliabilitydata retention policies, configured per data source

GeoAnalytics

Server

node 1

node 2

node 3node 4

node 5

r = 1

purge based on

data retention

spatiotemporal

big data store

GeoEvent

Server

Data reliabilityrolling index option, set appropriately to the velocity of your observation data

GeoAnalytics

Server

node 1

node 2

node 3node 4

node 5

r = 1

indices indices

indices indices

indices

purge based on

data retention

spatiotemporal

big data store

GeoEvent

Server

Data reliabilityautomatic data backups using periodic snapshots, including ability to restore from a snapshot

GeoAnalytics

Server

node 1

node 2

node 3node 4

node 5

r = 1

snapshot-2016-05-17-11-0-0.snapshot

snapshot-2016-05-17-12-0-0.snapshot

…

ObjectIdchoosing an Object Id option



Max Value # of IDs ArcGIS Clients

Int32 2,147,483,647 2.1 billion Pro, Desktop, Ops Dashboard, …

events per day Int32

1,000 e/s 86,400,000 25 days

10,000 e/s 864,000,000 2.5 days

100,000 e/s 8,640,000,000 6 hours

1,000,000 e/s 86,400,000,000 36 minutes

10,000,000 e/s 864,000,000,000 4 minutes




Int64 (signed) 9,223,372,036,854,775,807 9.2 quintillion JavaScript, custom apps

events per day Int32

1,000 e/s 86,400,000 25 days

10,000 e/s 864,000,000 2.5 days

100,000 e/s 8,640,000,000 6 hours

1,000,000 e/s 86,400,000,000 36 minutes

10,000,000 e/s 864,000,000,000 4 minutes





events per day Int32 Int64 (signed)

1,000 e/s 86,400,000 25 days 292,472,000 years

10,000 e/s 864,000,000 2.5 days 29,247,200 years

100,000 e/s 8,640,000,000 6 hours 2,924,720 years

1,000,000 e/s 86,400,000,000 36 minutes 292,472 years

10,000,000 e/s 864,000,000,000 4 minutes 29,248 years





UniqueStringID n/a unlimited JavaScript, custom apps

events per day Int32 Int64 (signed)

1,000 e/s 86,400,000 25 days 292,472,000 years

10,000 e/s 864,000,000 2.5 days 29,247,200 years

100,000 e/s 8,640,000,000 6 hours 2,924,720 years

1,000,000 e/s 86,400,000,000 36 minutes 292,472 years

10,000,000 e/s 864,000,000,000 4 minutes 29,248 years

Summary

http://links.esri.com/geoevent-sbds

spatiotemporal big data store

• The spatiotemporal big data store enables:

- GeoEvent Server to write high velocity & volume observation data

- On-the-fly aggregations to be visualized using Map & Feature Services

- GeoAnalytics Server to read & write high volume analytic results

• To learn more:

- See the ‘Spatiotemporal Big Data Store’ tutorial

Real-Time & Big Data GIS

• Applying Real-Time Analytics Mon, 1:45pm-2:45pm, Room 147B

Tue, 11:00am-12:00pm, Room 151B

• Leveraging the Spatiotemporal Big Data Store Mon, 3:00pm-4:00pm, Room 147B

Tue, 1:30pm-2:30pm, Room 151B

• ArcGIS GeoAnalytics: Intro to Big Data Feature Analytics Mon, 4:15pm-5:15pm, Room 147B

Tue, 2:45pm-3:45pm, Room 147A

• Leveraging Stream Services Tue, 2:45pm-3:45pm, Room 151A

• ArcGIS GeoEvent Server: Best Practices Tue, 4:00pm-5:00pm, Room 151A

• Real-Time & Big Data GIS at a Massive Scale Tue, 5:15pm-6:15pm, Room 151A

other sessions

Please Take Our Survey on the Esri Events App!

Select the session

you attended

Scroll down to find

the survey

Complete Answers

and Select “Submit”

Download the Esri Events

app and find your event

Print Your Certificate of Attendance

Print stations located in the 140 Concourse

Monday

12:30 PM – 6:30 PM

GIS Solutions Expo,

Hall B

5:15 PM – 6:30 PM

Expo Social,

Hall B

Tuesday

10:45 AM – 5:15 PM

GIS Solutions Expo,

Hall B

6:30 PM – 9:30 PM

Networking Reception,

Smithsonian National Air

and Space Museum

Questions / Feedback?

http://links.esri.com/geoevent

http://links.esri.com/geoevent-forum

To learn more:

Adam MollenkopfReal-Time & Big Data GIS Capability Lead, Esri

[email protected]

@amollenkopf

Suzanne FossReal-Time & Big Data Product Engineer, Esri

[email protected]

@sfoss_esri

mailto:[email protected]

Date post:	07-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

GeoEvent & GeoAnalytics - Esri€¦ · -the geohash spatial index only supports GCS WGS 1984,...

Documents