Unstructured Data Analysis in ArcGIS

James Jones - Esri

Julia Bell - Esri

Scott Graff - Microsoft

What is Unstructured Data?

• Does not have a recognizable structure or is loosely structured

• Can be in a variety of formats and storage mechanisms

• Word Documents

• Email

• Social Media Posts

• PowerPoint

• PDF

• Share drive

“Every two days we create as muchinformation as we did up to 2003”

Eric Schmidt, 2010

What does that look like?Every minute…

Twitter sees new 350,000 tweets Facebook has 510,000 comments posted, 293,000 statuses updated

15.2 million Text Messages are sent

954,000 new Microsoft Office documents are created

144 million e-mails are sent

How much spatial information are we missing out on?

How much spatial information are we missing out on?

How can we capture this information in ArcGIS?

CoordinatesCustom Locations

User defined keywords

Native Esri

Capability

What are you looking for?

What is the best tool?

How is it best used?

• Data is at least somewhat understood• Data benefits from identifiable and

repeating patterns• Little to no programming experience

available/needed

How to Integrate Unstructured Data into ArcGIS

ArcGIS Pro w/ LocateXT

ArcGIS Enterprise w/ LocateXT

ArcGIS Pro for Intelligence

Extract Locations from Unstructured Data

ArcGIS LocateXT

Extracting Locations with ArcGIS

• LocateXT Extension for ArcGIS Desktop and Enterprise

• Available for ArcMap 9.1 and later

• Available in ArcGIS Pro at 2.3

• 100% Feature function as ArcGIS Pro 2.4

• Uses pattern matching regular expressions (REGEX) to search for coordinates in a variety of formats

• Uses custom location list to match/extract other patterns (place names, codes, other terms)

Extracting Locations in ArcGIS Pro

• New option added to the “Add Data” button

• Allows for a user to drag and drop documents or copied text into a window

• Can create a new feature class or append it to an existing one

• Included with ArcGIS Pro for Intelligence

Extracting Locations in ArcGIS Pro

• Two Geoprocessing Tools added

• Located in the Conversion Tools –> To Geodatabase

• Extract Locations from Document

• Extract Locations from Text

Extracting Custom Attributes

• Ability to create custom attributes based on content within document or near a location

• Triggered by location extraction

• Based on keywords

• Tag locations based on keywords

• Scrape/harvest portions of document based on keywords

• Ability to extract based off of:

• Number of characters/words

• Number of lines/blank line

• Stop string

• Previously built in separate LocateXT desktop application (until Pro 2.4)

Extracting Addresses

• Ability to extract addresses from documents based on combination of:

• State

• Zipcode

• Ex. VA 22182

• The combination of extracted text and pre-text is geocoded

Explore Unstructured Data through LocateXT and Custom Attributes

How to Integrate Unstructured Data into ArcGIS

CoordinatesCustom Locations

User defined keywords

LocationsPeople/Organizations

EventsDates

Relationships

Native Esri

Capability

What are you looking for?

What is the best tool?

How is it best used?

• Data is at least somewhat understood• Data benefits from identifiable and

repeating patterns• Little to no programming experience

available/needed

• Data is not well understood• Data does not contain identifiable

and/or repeating patterns• Integration needed

Third Party

Integration

Natural Language Processing

ArcGIS Pro w/ LocateXT

ArcGIS Enterprise w/ LocateXT

ArcGIS Pro for Intelligence

with ArcGIS

Integrating NLP

Integrating NLP Capabilities with ArcGIS

• Many NLP offerings have Python APIs/SDKs or communicate over REST

• Integrates near seamlessly with ArcPy

• Create Python Toolboxes/Script Tools

• Allows to extract relevant data based on data local to their machine or as part of Enterprise Pipeline

• ArcGIS.Learn has incorporated support for Entity Recognition

APIs

Apps

Desktop

ArcGIS

NLTK

NLP Tools

Processing Unstructured Data Using ArcGIS and Microsoft Azure

Source Processing Storage Analysis Apps/Visualization

Key Take-aways:

1. Leverages modern, serverless processes and integration apps

2. Allows for a variety of NLP processes to be ran

3. Deep analytics with ArcGIS and Azure Cognitive Services/Machine Learning

Building an Unstructured Pipeline to Understand World Events

Source Processing Storage Analysis Apps/Visualization

Logic Apps / Power Automate watch RSS feeds and Website.Microsoft Cognitive Services extracts entities and analyze

sentiment.

Data is passed to ArcGIS GeoEvent Server for ingestion

into ArcGIS Platform. JSON files are stored in Azure Storage. Tabular data is

stored in a Azure SQL Data Warehouse.

ArcGIS Pro for Intelligence allows non-GIS Intel

professionals access to a custom experience of ArcGIS Pro to provide deep analysis. Microsoft Cognitive Services

further enriches data by running computer vision

against embedded images.

Operations Dashboard and ArcGIS Insights allows for very tailored views of the data to be

quickly analyzed and viewed by decision makers and non-

GIS Professionals.

ArcGIS Pro for Intelligence

Multi-int workstation for the intelligence professional

• Create and manage intelligence information

• Visualize and display your data in maps, charts, and timelines

• Perform spatial, temporal, relational, and predictive analysis

• Produce and disseminate intelligence products

NLP Integration with ArcGIS

Download the Esri

Events app and find your event

Select the session

you attended

Scroll down to

“Survey”

Log in to access the

survey

Complete the survey

and select “Submit”

Please Share Your Feedback in the App