11/18/2019
1
Geospatial Data Science & Artificial Intelligence
Rohit Singh
Session Overview
• What is AI, Machine Learning & Deep Learning
• Machine Learning
- What is Machine Learning?
- Machine Learning Tools in ArcGIS
• Spatial Data Science
- Notebooks for data science workflows
- Spatial analysis using ArcGIS API for Python
• Geospatial Deep Learning
- Types of models and their applications
- Training and deploying deep learning models
- Scalable deep learning with Image Server
• Geo-AI projects showcase
1
2
11/18/2019
2
Data Science
ArcGIS integrates all types of data, from various sources
Imagery
LidarVector
Tabular
Real-Time (IoT)
Big Data
Unstructured
BIM
3D
Network
Maps, Scenes, Layers
ArcGIS Enterprise
Supports interactive and batch analysis workflows
Scalable
Raster analysis Realtime analytics Big Data analysis GeocodingSpatial analysis Notebooks
Data Science
3
4
11/18/2019
3
• Clustering
• Prediction
• Classification
• Regression
• Interpolation
• Object Identification
New and Improved
• Feature Extraction
• Site Selection
• Event Prediction
• Image Analysis
Coming
Data Science, Machine Learning & Deep Learning
ArcGIS is a Data Science platform . . .and includes Machine Learning & Deep Learning
Empirical Bayesian KrigingRegression Prediction
Training Data Preparation
Density-BasedClustering
Forest-Based Classificationand Regression
Spatial Analysis
Transportation
Feature Identification
Deep Learning
Feature Extraction
Survey
Python Notebook Integration
ArcGIS
• PyTorch
• TensorFlow
• scikit-learn• Microsoft
• IBM Watson
• Amazon
R Integration
Data Science
SAS Jupyter
pandas
What is AI?
5
6
11/18/2019
4
Neural Networks
TensorFlow
CNTK
Natural Language Processing
Cognitive Computing
GeoAI
Computer Vision
Dimensionality Reduction
Object Detection
Support Vector Machines
Object Tracking
Keras
PyTorch scikit-learn
fast.ai
Random Forest Machine Learning
Deep Learning
Artificial IntelligenceCaffe
Machine Learning
Deep Learning
Artificial Intelligence
7
8
11/18/2019
5
Machine Learning
Artificial Intelligence
Deep Learning
Machine Learning
Artificial Intelligence
Deep Learning
Data Science
9
10
11/18/2019
6
Machine Learning
Labelled Data Sets: Features + Output
Call Drops
# Complains Subscribed Package
Call Rate Decline
Churned?
4 5 ABC 20% Yes
6 2 ABC 5% No
9 4 XYZ 12% Yes
Features Output (label)
Training Data(Historical)
For Learning..
New Data Prediction
Learning Models
Trained Model
Problem: Predicting Churn
7 5 KLM 8% Yes (75%)
12
13
11/18/2019
7
Labelled Data Sets: Features + Output
Segment TypeProximity to Intersection
Time of Day
Weather Accident
Highway 0.1 M Morning Raining Injury
Tunnel 0.3 M Evening Sunny Property
Inner 0.2 M Noon Foggy Injury
PipeAge
Depth Temperature Pressure Break
20 3 m 95 F 20 P Yes
15 4 m 5 F 35 P Yes
6 3.5 m 2 F 17 P Yes
IS Indirect Fire
IS Explosive Attack
IS Vehicle Attack
Coalition Air Ops
ISIS Camp
0.2 M 2 M 3 M 0.7 M Yes
4 M 9 M 6 M 1 M No
1.2 M 5 M 5 M 0.5 M Yes
# Females 35 - 50
F&B Sales% Urban
Chic#
CompetitorsSales
35,000 $400M 55% 15 $20M
14,000 $150M 15% 27 $5M
27,000 $210M 26% 9 $12M
Road Accidents Prediction Water Leakages Prediction
ISIS Location Prediction Retail Sales Prediction
Python API
R-Bridge
14
15
11/18/2019
8
Machine Learning in ArcGIS
ArcGIS has Machine Learning Tools
ArcGIS
Classification
Clustering
Prediction
16
17
11/18/2019
9
Using the known to estimate the unknown
Use Case: Accurately predict impacts of climate change on local temperature using global climate model data
Prediction
In ArcGIS: Empirical Bayesian Kriging, Areal Interpolation, EBK Regression Prediction, Ordinary Least Squares Regression and Exploratory Regression, Geographically Weighted Regression
The process of deciding to which category an object should be assigned based on a training dataset
Use Case: Classify impervious surfaces to help effectively prepare for storm and flood events based on the latest high-resolution imagery
Classification
In ArcGIS: Maximum Likelihood Classification, Random Trees, Support Vector Machine
18
19
11/18/2019
10
The grouping of observations based on similarities of values or locations
Use Case: Given the nearly 50,000 reports of traffic between 5pm and 6pm in Los Angeles (from Traffic Alerts by Waze), where are traffic zones that can be used to elicit feedback from current drivers in the area?
Clustering
In ArcGIS: Spatially Constrained Multivariate Clustering, Multivariate Clustering, Density-based Clustering, Image Segmentation, Hot Spot Analysis, Cluster and Outlier Analysis, Space Time Pattern Mining
The grouping of observations based on similarities of values
Use Case: Use different economic indicators to define economic regions automatically
Spatially-Constrained Clustering
Similar In ArcGIS: Multivariate Clustering, Density-based Clustering, Image Segmentation, Hot Spot Analysis, Cluster and Outlier Analysis, Space Time Pattern Mining
20
21
11/18/2019
11
Notebooks for data science workflows
What is ArcGIS Notebooks?
A new capability that lets you create hosted Python notebooks using a Juypterscripting environment that has been built into ArcGIS.
• Let’s break this down…
23
24
11/18/2019
12
Python is one of the most widely used programming languages. It is easy to learn and interpret and can be applied to many disciplines.
Python
Web GIS Administration
Content Management
Analysis and Data Science
Python + GIS use cases
Python Notebook / Jupyter
• Jupyter is open source software designed to support interactive data science and scientific computing.
• It is browser based and lets you write, run, and document your code all in one place.
• Esri has included Jupyter alongside ArcGIS Pro for several years and the ArcGIS API for Python was developed to make the most of Jupyter’s functionality.
25
26
11/18/2019
13
ArcGIS Notebooks sits at the intersection of ArcGIS and open data science
*Tools only available with a license of GeoAnalytics Server and Image Server, respectively
ArcGIS
Open SourcePython & Data Science
• ArcGIS API for Python• ArcPy• Analytics
• Geoprocessing • Web GIS analysis• GeoAnalytics*• Raster Analytics*
How is ArcGIS Notebooks different than “traditional” Jupyter?
• Built within ArcGIS rather than sitting beside it
• Leverages an integrated UI
- Automatically insert Python snippets
- Quickly access a gallery of samples
• Notebooks as items
• Use ArcPy without ArcGIS Pro
• Centrally manage libraries and packages
27
28
11/18/2019
14
- ArcGIS Pro
- R-ArcGIS Bridge, R Notebooks
- ArcPy Spatial machine learning
- Machine learning GP tools
- Deep learning tools for imagery and rasters
- ArcGIS Notebooks (coming soon)
- ArcGIS Enterprise
- GeoAnalytics Server
- Custom Python tools
- Machine learning tools
- Raster Analytics (Image Server)
- Deep learning
- Raster functions
- ArcGIS Notebook Server
- ArcGIS Online
- ArcGIS Notebooks (now in beta)
- ArcGIS API for Python
- arcgis.learn
Spatial data science across ArcGIS
ArcGIS Notebooks
29
30
11/18/2019
15
Features of ArcGIS Notebooks - Recap
• Jupyter interface is integrated with ArcGIS
- Use UI to browse for analysis tools and data and insert automatic Python snippets
• Access to content in your organization, the Living Atlas, and ArcGIS Online
• Create and run scripts with ArcPy
• Private workspace for data, interim results, and scratch environments
• Python libraries and packages are synchronized across the organization
- Customize what comes out of the box by modifying the notebook runtime
• Python notebooks are saved as items
- There is a new item type called Notebook
• Read-only previews of notebooks
• Secure identity-based access
• Esri Sample Notebooks
…and more
Esri sample notebooks
31
32
11/18/2019
16
Notebook Samples
• ArcGIS Notebooks includes end-to-end sample notebooks for:• Web GIS administration• Content management• Data Science and Analysis
• Samples can be used to learn something new or to jumpstart your own analysis
• Some samples are ready-to-run, while others require you to configure parameters
Spatial Analysis using Python
33
34
11/18/2019
17
Why Python?
• Popularity
• Productivity
• Interoperability
• Solves the “two-language” problem
• Scientific Python ecosystem
• Community
Source: Gateway Data Sciences Courses Reach Enrollment Milestones
35
36
11/18/2019
18
ArcGIS API for Python
gis
geometry
network
schematics
features
realtime
widgets
mapping
env
geocoding
geoenrichment
geoprocessing
raster
geoanalytics
Spatially Enabled
DataFrame
37
38
11/18/2019
19
39
40
11/18/2019
20
41
42
11/18/2019
21
Spatial Analysis Tools
• Summarize Data
• Find Locations
• Data Enrichment
• Analyze Patterns
• Use Proximity
• Manage Data
Summarize DataCalculate summary statistics for features and attributes
43
44
11/18/2019
22
Add detailed demographic data and statistics to your analysis
Enrich Layer
“What is near what?”
Use Proximity
45
46
11/18/2019
23
Use Proximity“What is near what?”
Manage DataManage geographic data, overlay layers
47
48
11/18/2019
24
Analyze PatternsIdentify, quantify, and visualize spatial patterns in your data.
Geocoding, Batch geocoding, reverse geocodingarcgis.geocoding module
Geocoding
49
50
11/18/2019
25
GeocodeSingle line or multi field addresses
GeocodePoints of Interest
• Cultural or geographic landmarks
• Businesses by name or category
• Administrative divisions
51
52
11/18/2019
26
Batch GeocodingGeocode an entire list of single line or multi field addresses
Reverse GeocodeDetermines address at a particular x/y location
53
54
11/18/2019
27
Routing and directions, location allocation, service areas…
arcgis.network.analysis module
Network Analysis
Network analysis capabilities and toolsCommercial grade, traffic aware routing and directions for multiple travel modes
55
56
11/18/2019
28
Network analysis – Service Areas
Enrich your analysis with demographic and business data
arcgis.geoenrichment module
GeoEnrichment
57
58
11/18/2019
29
GeoEnrichment
• Get facts about a location or area
- Street addresses
- Points, lines and polygon geometries
- Within a drive time or service area
- Named geographical areas
- Counties or block groups in California
- Districts and subdistricts in India
• Create charts and choropleth maps
GeoEnrichment – create reports
59
60
11/18/2019
30
Fast distributed spatio-temporal analysis of large vector and tabular data
arcgis.geoanalytics module
GeoAnalytics
GeoAnalytics Tools
• Summarize Data
• Manage Data
• Use Proximity
• Find Locations
• Analyze Patterns
61
62
11/18/2019
31
DemoAnalysis of crime patterns in Houston
63
64
11/18/2019
32
65
66
11/18/2019
33
Imagery and raster analysis
arcgis.raster module
Raster Analysis
67
68
11/18/2019
34
Image Processing
• Correction
• geometric and radiometric corrections
• the curved shape of the Earth,
• the imperfectly transparent atmosphere,
• daily and seasonal variations in the amount of solar radiation
• imperfections in scanning instruments, among other things.
• Visualization and Appearance
• brightness and contrast
• sharpen, blur, detects edges
• pansharpen
Image Processing (cont.)
• Band Math and Indices
• Image indices are computed from multiband images to emphasize a specific phenomenon.
• The normalized difference vegetation index (NDVI) is a standardized index allowing you to display greenness, also known as relative biomass.
• High values represent forest areas and lush vegetation.
• Moderate values represent areas of shrubs and grassland
• Very low values represent areas of little to no vegetation, such as concrete, rock, or bare soil.
• Extremely low or negative values represent areas with no vegetation at all, such as cloud, water, snow.
• The NDVI Colorized function applies the NDVI function on the input image, and then uses a color map or color ramp to display the result.
69
70
11/18/2019
35
Image Processing (cont.)
• Image Segmentation and Classification
• Perform object based image classification on image segments or pixels using a variety of supervised and unsupervised techniques through an easy-to-use wizard workflow.
• Supervised/unsupervised classification
• Object based and pixel based segmentation
• Accuracy assessment
• SVM, random trees, ISO clustering, MLC
Raster Analysis
• Distance Analysis
• distance, cost distance, shortest path, optimal travel path…
• Suitability Modeling
• Determine an ideal location for operations
• Combine categorial (land use, soil type) and continuous (slope, property value) variables to create a suitability map.
• Hydrological Analysis
• surface flow patterns, create watersheds, stream networks, flow distance, and other hydrological characteristics. Summarize landscape data as input to hydrologic and hydraulic models, as well as flood inundation and erosion models.
71
72
11/18/2019
36
Raster Analysis (contd.)
• Interpolation and Density
• Predicts values and creates a continuous surface from sample data points, such as elevation, rainfall, chemical concentrations, and noise levels.
• Derivation of land surface characteristics
• Create hillshade, slope, aspect, curvature, solar insolation, and visibility to help solve spatial problems
• Statistical Analysis
• Perform calculations on a per-cell basis between multiple datasets, based on local, neighborhood, or zonal functions.
• Generalization
• Either clean up small erroneous data in the raster or generalize the data to get rid of unnecessary detail for a more general analysis.
• Map Algebra
• An easy-to-use and powerful way to perform raster analysis as algebraic expressions.
Image Processing & Raster Analysis Functions
Math
CalculatorAbsDivideExpExp10Exp2FloatIntLn, Log10 Log2MinusModNegatePlusPowerRound DownRound UpSquareSquare RootTimes
ACosACosHASinASinHATanATan2ATanHCosCosHSinSinHTanTanH
ConSet Null
Data Management & Conversion
Raster to VectorVector to RasterColormapColormap To RGBComplexGrayscaleRemap / Reclass Spectral ConversionUnit Conversion Vector FieldLAS to RasterLAS Dataset to RasterClip CompositeExtract BandsMaskMosaic RastersRasterize FeaturesReprojectNibble
Interpolation
Interpolate Irregular DataNearest NeighborIDWEBKSwath
Correction
Apparent ReflectanceGeometric CorrectionSpeckle Filtering (Lee,Frost,Kuan)Thermal NoiseRadiometric Calibration
Analysis: Image Segmentation & Classification
Segmentation (Mean Shift)Training (ISO, SVM, ML)Supervised Classification
Visualization & Appearance
Contrast and Brightness Convolution Pansharpening Resample Statistics and Histogram Stretch
Surface Generation & Analysis
Aspect Curvature Elevation Void Fill Hillshade Shaded Relief Slope ViewshedContour
Analysis: Overlay
Weighted OverlayWeighted Sum
Analysis: Band Math & Indices
NDVI / NDVI Colorized SAVI / MSAVI / TSAVIGEMIGVI (Landsat TM)PVITasseled Cap (Kauth-Thomas)Binary Thresholding
Analysis: Distance & Density
Euclidean Distance Cost DistanceLeast Cost PathKernel Density
Statistics:
Zonal StatisticsCell StatisticsArgStatistics
Python
Custom Algorithms
Analysis: Hydrology
FillFlow AccumulationFlow DirectionFlow DistanceStream LinkWatershed
Multiband Math
Arithmetic Band Arithmetic
BitwiseAndLeft ShiftNotOrRight ShiftXor
BooleanAndNotOrXor
Equal ToGreater ThanGreater Than EqualIs Null Less ThanLess Than EqualNot Equal
73
74
11/18/2019
37
Applications
Effects of Climate Change
Clark Labs
Change Analysis
DigitalGlobe
Forest Acidification
Midwestern US Loss of Habitat
Central Africa
Blue Raster
Farm Management
New Zealand
Crop Rotation
USA
FEMA
Assessing Vulnerable Populations
Wildfire Risk
FranceCaroviz
Volcanic Monitoring
JapanJAXA
Watersheds
arcgis.learn moduleDeep Learning using ArcGIS Python API
75
76
11/18/2019
38
arcgis.learn
The arcgis.learn module in ArcGIS API for Python enables GIS analysts and data scientists to easily adopt and apply deep learning in their workflows.
It enables training state-of-the-art deep learning models with a simple, intuitive API.
Beyond Detections: End to End GeoAI Lifecycle enabled by ArcGIS Imagery AI Capabilities
ImageryAccess
Imagery Prep
Training Data Prep
Train & ConsumeModels
Run Inference at SCALE
Feedback Loop
TakeAction
Deploy Models to Production
Before After
• Installing External DL
Frameworks
• Dozens of lines of
Code
• HARD!
• No Installation
(Notebooks)
• 3-5 lines
• EASY
77
78
11/18/2019
39
Not just “Training”!
Exporting Training Data arcgis.learn.export_training_data
Training DL Modelsarcgis.learn.SingleShotDetectorarcgis.learn.UnetClassifierarcgis.learn.FeatureClassifierarcgis.learn.PSPNetClassifierarcgis.learn.RetinaNetarcgis.learn.MaskRCNNarcgis.learn.EntityRecognizer
Preparing Data (Augmentation)arcgis.learn.prepare_data
Model Managementarcgis.learn.list_models
arcgis.learn.Model
Model.install
Model.uninstall
Model.query_info
Inference APIsarcgis.learn.detect_objects
arcgis.learn.classify_pixels
Things you can do today with ArcGIS.LearnObject Detection using SSD, Pixel Classification using Unet, Feature Classification
Damaged Structures
Roads
Swimming Pools
Building Footprints
Oil Pads
Land Cover
Road Cracks
Cars Palm Trees
Pipeline Encroachment
79
80
11/18/2019
40
Geospatial Deep Learning
81
82
11/18/2019
41
Why use Deep Learning for Imagery
ImageNet Visual Recognition Challenge error rate
• Better than human accuracy at vision tasks
Deep Learning Workflowin ArcGIS
83
84
11/18/2019
42
Train DL ModelTrain DL Model
Detect ObjectsDetect Objects
Export
Training
Data For DL
Export
Training
Data For DL ModelModel
Training SamplesTraining Samples
ArcGIS Pro ArcGIS Proarcgis.learn
Classify Pixels
Collect Samples Export Training
SamplesTrain
Perform Inference
ArcGIS Enterprise ArcGIS Enterprise
ArcGIS Deep Learning Workflow
Data Labeling: Training Samples Manager
• Add Labels
• Quickly Collect Samples
• Save Samples to a Feature Class
Collect Samples
Export TrainingSamples
Train Perform Inference
85
86
11/18/2019
43
Export Training Data for Deep Learning Tool
Collect Samples
Export TrainingSamples
TrainPerform Inference
• Exports Samples to Training Images• Each Image has Labels• Supports various formats
Train Model
• arcgis.learn module in ArcGIS API for Python
- No installation (ArcGIS Notebooks)
- Easy to use - 3-5 lines of code
- Models:
- Object Detection
- Pixel Classification
- Feature Classification
• External Deep Learning Frameworks
- Tensorflow
- PyTorch…
Collect Samples
Export TrainingSamples
Train Perform Inference
87
88
11/18/2019
44
Consume Deep Learning Models
Perform Inference
Model DefinitionModel Definition
ArcGIS UserArcGIS User
Inference results
Inference results
Input ImagesInput Images
InferenceTools
InferenceTools
• ArcGIS Image Analyst in Pro
• ArcGIS Image Server on Enterprise
Inference ToolsInference Tools
• Classify Pixels Using Deep Learning
• Object Detection Using Deep Learning
Non Maximum Suppression
Collect Samples
Export TrainingSamples
Train Perform Inference
Classify Pixels Using Deep Learning
Runs the model on an input raster to product a classified
raster, each valid pixel has an assigned class label.
• Mini-batch support
• Processor type: CPU or GPU
• Parallel processing in ArcGIS Pro and distributed
raster analysis on Enterprise
Inference Tools
• ArcGIS Image Analyst in Pro
• ArcGIS Image Server on Enterprise
89
90
11/18/2019
45
Detect Objects Using Deep Learning
Runs the model on an input raster to produce a feature class
containing the objects it finds.
• Batch processing
• Optional Non Maximum Suppression
• Processor type: CPU or GPU
• Parallel processing in Pro and distributed raster analysis
on Enterprise
Inference Tools
• ArcGIS Image Analyst in Pro
• ArcGIS Image Server on Enterprise
ArcGIS in use for each step of the deep learning workflow
End-to-end from raw imagery to structured information products
ArcGIS Deep Learning Workflow
Imagery Labelling Data Prep
Train Model
Detect Objects
AnalysisField
Mobility, Monitoring
92
93
11/18/2019
46
Computer Vision TasksApplied to GIS
Semantic SegmentationAssign a label to each pixel
Cat
Ground
Sky
Turf/Grass
Building
Water
Pixel Classification
Applications:
- Land Cover Classification
- Pervious/Impervious mapping…
Model Architectures:
- U-Net
- PS Net…
94
95
11/18/2019
47
Object DetectionFind objects and their location (bounding boxes)
Applications:
- Detect trees, cars, airplanes, …
Model Architectures:
- Single Shot Detector (SSD)
- YOLO
- RetinaNet
Image ClassificationAssign a label to a given image
Cat
Applications:
- Damaged building classification
- Clean or ‘green’ pools…
Undamaged Damaged
Feature ClassificationAssign a label to a given feature
Model Architectures:
- Inception
- ResNet
- VGG…
96
97
11/18/2019
48
Instance SegmentationFind objects and their precise locations (masks or polygonal features)
Applications:
- Building footprint extraction
Model Architectures:
- Mask-RCNN
- DeepMask…
Impervious Surface Classification
Coconut Tree Detection
Building Footprint Extraction
Damaged House Classification
Applications of Deep Learning to Imagery
Pixel Classification Object Detection Instance Segmentation Image Classification
End to End Deep Learning – Wide spectrum of deep learning models
98
99
11/18/2019
49
Scalable Deep Learning
60,000 buildings
ArcGIS Pro: 1 GP100 GPU (16 GB):
4.5 hours
ArcGIS Enterprise:
4 nodes RA server with 3 x P40
GPUs (24GB):
20 minutes
100
101
11/18/2019
50
Geo-AI projects showcase
ETA PredictionPredict Arrival Times accurately to Optimize Supply Chain & Boost Customer Satisfaction
102
103
11/18/2019
51
What if we want to accurately Predict Arrival Times per Fleet Car, taking into consideration:
• Weekend VS Weekday
• Winter VS Summer
• Holidays VS Traditional Days
• Rush-Hour VS Traditional Hours
• Type of Vehicle
• Time of the Day
• Type of Products
• .. Other variables
Traditional Rout-Solvers wan’t do the job, we need Machine Learning to:
• Mine the deep patterns in historical delivery data (Origin, Destination, Total Time, Time o the Day, Vehicle Type, Weather, Type of Products..etc)
• Build Predictive Models that take such seasonality and patterns into consideration, to predict arrival times with high accuracy
ETA Prediction
Nvidia GPU:
2560 CUDA cores
300K Predicted
Points/Second
(VS 4-5 points/sec NA)
Network Analyst (Blue) VS Predictions (Red)
Training Data: 320M Network Analyst OD Points
104
105
11/18/2019
52
106
107