GeoEvent Server: Best Practices · 2017-03-14 · Box REST.csv WS WS HTTP i y ActiveMQ CAP...

GeoEvent Server:Best Practices

Adam MollenkopfReal-Time & Big Data GIS Capability Lead, Esri

[email protected]

@amollenkopf

Suzanne FossReal-Time & Big Data Product Engineer, Esri

[email protected]

@sfoss_esri

mailto:[email protected]

1

2

3

4

Agenda

ingestion & analytics

dissemination & visualization

storage

resilience & scalability

ingestion & analytics1

you can create

your own

connectors

Ingestionuse an existing input connector

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Poll an ArcGIS Server for Features

Poll an external website for GeoJSON, JSON, or XML

Receive Features, GeoJSON, JSON, or XML on a REST endpoint

Receive RSS

Receive GeoJSON or JSON on a WebSocket

Receive Text from a TCP or UDP Socket

Subscribe to an external WebSocket for GeoJSON or JSON

Watch a Folder for new CSV, GeoJSON, or JSON Files

Ou

t o

f th

e B

ox

REST

.csv

WS

WS

HTTP

Esri

Gallery

ActiveMQ

CAP

Instagram

Exploitation Support Data

Cursor-on-Target

RabbitMQ

NMEA 0183

MQTT

Sierra Wireless (RAP)

KML

Kafka

Trimble (TAIP)

Twitter

Part

ner

Gallery

CompassLDE

enviroCar

GNIP

FAA (ASDI)

exactEarth AIS

Waze

Valarm

Networkfleet

OSIsoft *

*

*

*

Zonar *

Amazon IoT

Azure IoT

you can create

your own

connectors

Ingestionconfigure a new input connector by pairing a transport & adapter together

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Feature-JSON

Connectors Transport Adapter

GeoJSON

Receive Feature-JSON from Kafka

Receive GeoJSON on a REST endpoint

Feature-JSON

Adapters

GeoJSON

JSON

RSS

Text

XML

CAP

Cursor-on-Target

GeoMessage

Instagram

NMEA

Sierra Wireless (RAP)

Trimble (TAIP)

Twitter

VMF

Es

riG

all

ery

Waze

Feature Service

Transports

File

HTTP

HTTP+BasicAuth

HTTP+OAuth

TCP

UDP

Waze

WebSocket

Esri

Gallery Amazon IoT

Azure IoT

Kafka

IRC

Kafka

MQTT

Kafka

HTTP

RabbitMQ

Twitter

Ingestionclone an existing connector & tailor it’s properties to make a more specific connector

Ingestionclone an existing connector & tailor it’s properties to make a more specific connector

Ingestionextend GeoEvent by downloading additional transports & adapters

https://github.com/esri/solutions-geoevent-java

http://links.esri.com/geoevent-gallery

AnalyticsGeoEvent Services best practices

input1 output1buffer

• Which is better?

- have fewer but more complex GeoEvent Services

- or have more but simpler GeoEvent Services

geofences

filter(spatial)

input1 output2

geofences

filter(spatial)

geofences

geotagger

input2 output3

geofences

filter(spatial)

motioncalculator

geometry inside Zones/.*

geometry inside Zones/.* geometry inside Zones/.*


AnalyticsGeoEvent Services best practices

• Which is better?

- have fewer but more complex GeoEvent Services

- or have more but simpler GeoEvent Services

input1 output1buffer

geofences

filter(spatial)

input2 output2

geofences

geotagger

output3motion

calculator



dissemination & visualization2

you can create

your own

connectors

Disseminationuse an existing output connector

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Ou

t o

f th

e B

ox

Add or Update a Feature

Publish Text to a UDP Socket

Send a Text Message

Send an Email

Push Text to an external TCP Socket

Push GeoJSON or JSON to an external WebSocket

Push GeoJSON or JSON to an external Website

Send an Instant Message

Send Features to a Stream Service

Write to a CSV, GeoJSON, or JSON File .csv

WS

im

HTTP

Add a Feature to a spatiotemporal big data store

Update a feature in a spatiotemporal big data store

ActiveMQ

Esri

Gallery

Cursor-on-Target

Hadoop

Kafka

MongoDB

MQTT

RabbitMQ

Twitter

Amazon IoT

Azure IoT

you can create

your own

connectors

Disseminationconfigure a new output connector by pairing an adapter & transport together

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Message Formatter SMTP

Connectors TransportAdapter

Feature-JSON Kafka

Send an Email

Publish Feature-JSON to Kafka

Big Data Store

Transports

Feature Service

File

HTTP

HTTP+BasicAuth

HTTP+OAuth

SMTP

Stream Service

TCP

UDP

WebSocket

XMPP

Cache

Adapters

Feature-JSON

GeoJSON

JSON

Message Formatter

RSS

Text

WebSocket

ActiveMQ

Amazon IoT

Azure IoT

Hadoop

IRC

Kafka

Esri

Gallery

MongoDB

MQTT

RabbitMQ

Cursor-on-Target

Esri

Gallery

SMS

TCP-Squirt

Twitter

you can create

your own

connectors

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Ou

t o

f th

e B

ox


Publish Text to a UDP Socket

Send a Text Message

Send an Email

Push Text to an external TCP Socket

Push GeoJSON or JSON to an external WebSocket

Push GeoJSON or JSON to an external Website

Send an Instant Message

Send Features to a Stream Service

Write to a CSV, GeoJSON, or JSON File .csv

WS

im

HTTP

ActiveMQ

Esri

Gallery

Cursor-on-Target

Hadoop

Kafka

MongoDB

MQTT

RabbitMQ

Twitter

Add a Feature to a spatiotemporal big data store

Update a Feature in a spatiotemporal big data store

Visualizationvisualizing real-time observation results using Esri clients

Visualizationchoosing a service type: stream service, feature service, map service

Stream Layer

Map Layer

Feature Layer

• Stream layers in apps subscribe to stream services to immediately visualize observations

- does not require storage, low latency, no playback

• Map & Features layers in apps periodically poll to visualize most current observations

- backed by an enterprise geodatabase (EGDB) or a spatiotemporal big data store (BDS)

- history can be retrieved & queried for playback

desktop

device

web

ArcGIS

Enterprise

Stream Service

Send Features to a Stream Service subscribe (push)

polling (pull)

Map Service

Feature Service

GeoAnalytics

Server

Add a Feature to a BDS

Update a Feature in a BDS

spatiotemporal

big data store


EGDBGeoEvent

Server

Visualizationworkflow for creating a real-time service using stream service, feature service, map service

• Configure an input to receive real-time data & define a geoevent definition accordingly

• Create an output and publish a Stream / Feature / Map Service using the geoevent definition

• Author and publish a geoevent service

• Visualize your Stream / Feature / Map Service

ArcGIS Enterprise

spatiotemporal

big data storeGeoEvent

Server

input

geoevent

definition

geoevent service

Stream Service

Map Service

Feature Service

output

storage3

ArcGIS Enterprisewith real-time & big data capabilities

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big DataIoT

GeoAnalytics

Server


ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big DataIoT

GeoAnalytics

Server

1

2 43

MINIMUM environment

functional servers & spatiotemporal big data store

SHOULD BE on ISOLATED machines!!!


ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big DataIoT

GeoAnalytics

Server

1

2 3 64 75 8

RECOMMENDED environment

functional servers & spatiotemporal big data store

SHOULD BE on ISOLATED machines!!!

resource minimum RECOMMENDED notes

64-bit

operating system

(Windows)

Windows Server:

2012 R2 Standard & Datacenter (Sep 2016)

2012 Standard & Datacenter (Sep 2016)

2008 R2 Standard, Enterprise & Datacenter (sp1)

2008 Standard, Enterprise & Datacenter (sp2)

Windows Server:

2012 R2 Standard & Datacenter (Sep 2016)

‘_’ characters are

not allowed in

machine names

64-bit

operating system

(Linux)

Red Hat Enterprise Linux Server 7 (update 8)

Red Hat Enterprise Linux Server 6 (update 2)

CentOS Linux 7 (7.2)

CentOS Linux 6 (6.8)

Scientific Linux 7 (7.2)

Scientific Linux 6 (6.8)

Ubuntu Server LTS (16.04.1)

SUSE Linux Enterprise Server 12 (sp1)

SUSE Linux Enterprise Server 11 (sp4)

Oracle Linux 7 (update 2)

Oracle Linux 6 (update 8)

Use your Linux flavor of choice with the

latest update version Esri has certified.

http://server.arcgis.com/en/data-store/latest/install/windows/arcgis-data-store-system-requirements.htm

http://server.arcgis.com/en/data-store/latest/install/linux/arcgis-data-store-system-requirements.htm

spatiotemporal big data storeoperating system requirements

resource requirements: processors




processors 8 cores 16 cores

More processors results in

faster writes (indexing time),

higher write throughput &

faster query response time.

spatiotemporal big data store

resource requirements: processors, memory









storage 400 GB 1 TB or moreStorage need is dictated by data velocity & size.

SSD results in much faster writes & queries.


resource requirements: processors, memory, storage











memory 16 GB 32 GBMore memory results in

higher likelihood of query cache hits.

32 GB is the maximum due to JVM limitations.


resource requirements: processors, memory, storage & instances









memory 16 GB 32 GBMore memory results in

higher likelihood of query cache hits.

32 GB is the maximum due to JVM limitations

# of

ISOLATED

instances

1 3 or more

With one instance there is risk of data loss.

With three instances replicas are written

increasing data reliability.

The # of instances will vary based on the velocity of

incoming data as well as the data retention policy

configured on each data source.




spatiotemporal

big data store

GeoEvent

Server

shards & replication factor

GeoAnalytics

Server

node 1

node 2

node 3node 4

node 5

T1

T3

T2

T2

r = 1

T1

T3


spatiotemporal

big data store

auto-rebalancing of data upon node membership changes, + or -, in the big data store

GeoEvent

Server

GeoAnalytics

Server

node 2

node 3node 4

node 5

T1

T3

T2

T2

r = 1

x

T1

T3

T1

T1T3


purge based on

data retention

spatiotemporal

big data store

GeoEvent

Server

data retention policies, configured per data source

GeoAnalytics

Server

node 1

node 2

node 3node 4

node 5

r = 1


purge based on

data retention

spatiotemporal

big data store

GeoEvent

Server

rolling index option, set appropriately to the velocity of your observation data

GeoAnalytics

Server

node 1

node 2

node 3node 4

node 5

r = 1

indices indices

indices indices

indices


purge based on

data retention

spatiotemporal

big data store

GeoEvent

Server

automatic data backups using periodic snapshots, including ability to restore from a snapshot

GeoAnalytics

Server

node 1

node 2

node 3node 4

node 5

r = 1

snapshot-2016-05-17-11-0-0.snapshot

snapshot-2016-05-17-12-0-0.snapshot

…


choosing an Object Id optionspatiotemporal big data store

choosing an Object Id optionspatiotemporal big data store

choosing an Object Id option

Max Value # of IDs ArcGIS Clients

Int32 2,147,483,647 2.1 billion Pro, Desktop, Ops Dashboard, …

events per day Int32

1,000 e/s 86,400,000 25 days

10,000 e/s 864,000,000 2.5 days

100,000 e/s 8,640,000,000 6 hours

1,000,000 e/s 86,400,000,000 36 minutes

10,000,000 e/s 864,000,000,000 4 minutes





Int64 (signed) 9,223,372,036,854,775,807 9.2 quintillion JavaScript, custom apps

events per day Int32

1,000 e/s 86,400,000 25 days

10,000 e/s 864,000,000 2.5 days

100,000 e/s 8,640,000,000 6 hours

1,000,000 e/s 86,400,000,000 36 minutes

10,000,000 e/s 864,000,000,000 4 minutes






events per day Int32 Int64 (signed)

1,000 e/s 86,400,000 25 days 292,472,000 years

10,000 e/s 864,000,000 2.5 days 29,247,200 years

100,000 e/s 8,640,000,000 6 hours 2,924,720 years

1,000,000 e/s 86,400,000,000 36 minutes 292,472 years

10,000,000 e/s 864,000,000,000 4 minutes 29,248 years






UniqueStringID n/a unlimited JavaScript, custom apps

events per day Int32 Int64 (signed)

1,000 e/s 86,400,000 25 days 292,472,000 years

10,000 e/s 864,000,000 2.5 days 29,247,200 years

100,000 e/s 8,640,000,000 6 hours 2,924,720 years

1,000,000 e/s 86,400,000,000 36 minutes 292,472 years

10,000,000 e/s 864,000,000,000 4 minutes 29,248 years


resilience & scalability4

best practices

000

GeoEvent resilience & scalability

• GeoEvent clustering was introduced at 10.3

• However, observed to be brittle and subject to failure:

- Required at least 3 nodes in a cluster

- when a node goes down, it must be brought back up ASAP.

- a 2-node cluster can cause a split-brain condition in cluster.

- Must be physical hardware

- many customers only have virtualized environments.

- Network must be stable

- network instabilities can result in intermittent membership.

• Given this, GeoEvent clustering

- has been deprecated at the 10.5 release

- should be avoided and is discouraged

• Instead, if resiliency or addt’l scalability is required:

- isolated deployments should be employed

- this is true at 10.3, 10.4 & 10.5

isolated deployment

• Each GeoEvent instance exists in it’s own site and shares a common configuration manually.

• A message broker must be stood up in front to provide a common event source.

• GeoEvent instances share a common message broker and run active/active competing to

consume events.


bring-your ownmessage broker

isolated deployment

planes

trains

1 1

1 1



isolated deployment

planes

trains

2

2 2

2



isolated deployment

planes

trains

1 1

1 1



isolated deployment

• Known constraints:

- does not support failover of state: enter, exit, track gap detection, idle detector

- project team/IT staff needed to setup/configure message broker and get all sensor events into it

- input to GeoEvent is the message broker only, lose flexibility of all the input connectors available



isolated deployment

• An isolated deployment of GeoEvent instances leads to challenges with Stream Services:

- Client A & B see event 1, while client C & D do not

- Client C & D see event 2, while client A & B do not

1

1 1

1

1

2

2

2

2

2

Stream Service resilience & scalability


isolated deployment

• Using Kafka the GeoEvent instances can be configured to use separate consumer groups:

- With this configuration, all clients see all events

- note: can also be accomplished with RabbitMQ fan-outs or ActiveMQ topics

1

1 1

1

1

1 1

1

1



isolated deployment




2

2

2

2

2

2 2

2

2

1

1

1

1



isolated deployment

• A reverse proxy can be configured in between the clients and the stream services so that clients

don’t have direct knowledge of the servers they are connecting to.

- NGiNX is one such reverse proxy.

1

1 1

1 1

1

1

1

1


bring-your ownmessage broker bring-your own

reverse proxy

isolated deployment




2

2

2

2 2

2

2

1

1

1

1


bring-your ownmessage broker bring-your own

reverse proxy

How can we

simplify this?

Resilience & scalabilityof real-time capabilities @ 10.5

ArcGIS

Enterprise

spatiotemporal

big data store

GeoEvent

Server

4K e/s

4K e/s

4K e/s

Big Data

GeoAnalytics

Server



ArcGIS

Enterprise

spatiotemporal

big data store4K e/s

4K e/s

4K e/s

Big Data

GeoAnalytics

Server

connect, event hub &

GeoEvent Server


ArcGIS

Enterprise

spatiotemporal


4K e/s

4K e/s

Big Data

GeoAnalytics

Server

connectevent hub &

GeoEvent Server


ArcGIS

Enterprise

spatiotemporal


4K e/s

4K e/s

Big Data

GeoAnalytics

Server

event

hubconnect

GeoEvent

Server

SummaryGeoEvent Server: best practices

• When applying Real-Time GIS there are many best practices for

- ingestion & analytics

- dissemination & visualization

- storage

- resilience & scalability

• To learn more:

- See the tutorial:

‘ArcGIS GeoEvent Server: Resiliency’

coming soon

Real-Time & Big Data GIS

• Developing Real-Time Web Apps with JavaScript Thu, 3:00-3:30pm, Demo Theater 1, Oasis 1

• Big Data: Leveraging the Spatiotemporal Big Data Store Thu, 4:00-5:00pm, Catalina/Madera

• Building Android Location Awareness with GeoEvent Server Thu, 6:00-6:30pm, Mesquite C

• GeoEvent Server: Making 3D Scenes Come Alive Fri, 8:30-9:30am, Primrose C-D

• GeoEvent Server: Internet of Things (IoT) Fri, 1:00-2:00pm, Primrose

other sessions

Questions / Feedback?

http://links.esri.com/geoevent

http://links.esri.com/geoevent-forum

To learn more:

Adam MollenkopfReal-Time & Big Data GIS Capability Lead, Esri

[email protected]

@amollenkopf

Suzanne FossReal-Time & Big Data Product Engineer, Esri

[email protected]

@sfoss_esri

mailto:[email protected]

Date post:	28-Jul-2020
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

GeoEvent Server: Best Practices · 2017-03-14 · Box REST.csv WS WS HTTP i y ActiveMQ CAP...

Documents