Outline
• Scientific multidimensional data in ArcGIS• Reading, managing, visualizing and exploring• Performing analysis• Extending analytical capabilities using Python• Sharing as services and consuming• Resources
Scientific Multidimensional Data in ArcGIS
• Multidimensional Data• Observed data
Monitoring stations, other platformsRemotely sensed
• Reconstructed data• Simulation Model results
• Examples• Ocean data
Sea temperature, salinity, ocean current
• Weather dataTemperature, humidity, wind
• LandSoil moisture, NDVI, land cover
• Formats• NetCDF, GRIB, and HDF • Esri Mosaic dataset and Cloud Raster Format (CRF)
ArcGIS
direct ingest
data management
visualizationanalysis
share
Scientific Multidimensional Data in ArcGIS: Vision
• Spatial, temporal, and vertical dimensions
• Contains one or multiple variables in one file- A variable is cube or cubes- A variable has a time and/or depth- Each slice is a 2D array
• Multidimensional Raster- A Mosaic Dataset- Or a Raster Dataset
Multidimensional Raster
141 241 341 441
131 231 331 431
121 221 321 421
111 211 311 411
142 242 342 442
132 232 332 432
122 222 322 422
112 212 312 412
143 243 343 443
133 233 333 433
123 223 323 423
113 213 313 413
t = 1t = 2
t = 3
time
longitude
Climate and Forecast (CF) Conventionhttp://cfconventions.org/
Initially developed for• Climate and forecast data• Atmosphere, surface and ocean model-generated data• Also for observational datasets
• CF is now the most widely used conventions for geospatial netCDFdata. It has the best coordinate system handling.
• Current version 1.8
• You can use Compliance checker utility to check a netCDF file. http://cfconventions.org/compliance-checker.html
CF Convention
• Directly reads netCDF, HDF and GRIB file using - Multidimensional Raster Layer option from Add Data- Multidimensional tools
- Make Multidimensional Raster Layer- Make NetCDF Raster Layer (netCDF)- Make NetCDF Feature Layer (netCDF)- Make NetCDF Table View (netCDF)
• Reads OPeNDAP Service- Make OPeNDAP Raster Layer- Output dynamic multidimensional raster- Support Sub-setting
• NetCDF, HDF and GRIB formats are supported in mosaic dataset
Reading Scientific data in ArcGIS
• Creates a seamless multi-dimensional cube from- Performs spatial and temporal aggregation - Files representing different regions- Files representing different time steps/slices
• Mosaic Dataset- Dynamic mosaic of imagery collection- Supports multiple files and variables, normalize time and depth - Table of records- Lives in Geodatabase- A Raster Dataset- Supports On-the-fly analysis- Supports sharing as servicesCreateMosaicDataset(in_workspace, in_mosaicdataset_name, coordinate_system, {num_bands}, {pixel_type}, {product_definition}, {product_band_definitions})AddRastersToMosaicDataset(in_mosaic_dataset, raster_type, input_path, {update_cellsize_ranges}, {update_boundary}, {update_overviews}, {maximum_pyramid_levels}, {maximum_cell_size}, {minimum_dimension}, {spatial_reference}, {filter}, {sub_folder}, {duplicate_items_action}, {build_pyramids}, {calculate_statistics}, {build_thumbnails}, {operation_description}, {force_spatial_reference}, {estimate_statistics}, {aux_inputs}, {enable_pixel_cache}, {cache_location})
Multidimensional Raster Data Management using Mosaic Dataset
Table
Mosaic
Raster
Mosaic Dataset
Raster Datasets
Raster Datasets
• Slicing• Multidimensional Ribbon in Pro• Select By Dimension Tool• Layer Property dialog in ArcMap
• Temporal animation using Time Slider• Dimensional animation using Range Slider
• Predefined renderer• Vector Field to display wind speed or ocean current
• Adding a custom Scientific Color Scheme to your ArcGIS Pro Project
Visualizing Multidimensional Data
• Temporal Profile• Visualize the distribution of a variable over time
• Summary Statistics• Compute summary statistics of a variable
Exploring Multidimensional Raster
Methods
PropertiesRaster Object
RasterObject
• properties• catalogPath• path • name • isTemporary• readOnly
• isInteger• hasRAT• isMultidimensional• bandCount• bandNames
• extent • width• height • meanCellWidth• meanCellHeight• spatialReference
• maximum• mean• minimum • standardDeviation• noDataValue• noDataValues
• mdinfo• variableNames• variables • slices
• format• pixelType• compressionType• uncompressedSize• blockSize
• save• exportImage
• getProperty• setProperty• getRasterInfo• getRasterBands
• getVariableAttributes• renameVariable• removeVariables
• getDimensionNames• getDimensionAttributes• getDimensionValues• addDimension
• read• write
Create from a raster dataset
Create an empty raster from rasterInfo
Output from a map algebra expression Raster (inRaster, {is_Multidmensional})
Multidimensional Raster Properties and Methods
Properties• mdinfo• variableNames• variables • slices
Methods: Variable• getVariableAttributes• renameVariable• removeVariables
Methods: Dimension• getDimensionNames• getDimensionAttributes• getDimensionValues• addDimension
Methods: Variable• getVariableAttributes (variable_name)• renameVariable (current_variable_name, new_variable_name)• removeVariables (variable_names)
Methods: Dimension• getDimensionNames (variable_name)• getDimensionAttributes (variable_name, dimension_name)• getDimensionValues (variable_name, dimension_name)• addDimension (variable, new_dimension_name, dimension_value, {dimension_attributes})
time
longitude
ArcPy – Over 1300+ Comprehensive GIS ToolsData Management
• Convert between formats, 2D <-> 3D, build topology, data reviewer, linear referencing,
• data comparison, manage spatial databases, distributed editing, servers & services
• Specialized datasets –parcels, utility networks, address locators, network datasets
Raster Analysis• Extraction, overlay, proximity,
distance, hydrology, multivariate, map algebra, neighborhood, fuzzy overlays
Spatial Machine Learning
• Interpolations, spatial sampling, simulations,
• Stats – enrich, density, hot spot, clustering, autocorrelations, balanced zones, outliers, similarity search, local bivariate relationships
• Prediction – random forests, GLR, OLS, GWR, validation
• Pattern mining – emerging hot spot, time series clustering,
• Big Data – GeoAnalyticsdesktop tools
• Deep Learning – classify objects, classify pixels, detect objects, export training data, train model
3D• LAS classification, extraction,
terrain, TIN, DEM, DSM, volume analysis
• Surface analysis – aspect, contour, cut fill, hillshade, slope, line of sight, sun shadow, viewshed, hydrology
Location analytics
• business intelligence –market penetration, Huff model, suitability analysis, territory design,
• Network – routing, OD cost matrix, service area, location allocation
• Geocoding – batch, reverse, composite
Cartography• annotations, generalization,
map series, masks• 100+ projections and
transformations• editing tools – conflation,
densify, cartographic refinement
• Layouts, printing, map series
Imagery analysis
• management – mosaic datasets, raster algebra,
• ortho mapping – point cloud, stereo model, photogrammetry, surface creation, ortho rectification
• information extraction –classification, object detection, segmentation, spectral analysis, statistics
• Full motion imagery, multidimensional rasters, solar radiation
• Hundreds of math and stat tools
Multidimensional Analytical Tools Data aggregation
Anomaly calculation
Trend calculation
Finding statistics
Predict
• Several analytical functions are available out of the box• Functions are chained together to create complex model• Used to perform on-the-fly analysis• Extend analytical capability using Python Raster Function• Can be saved as a Raster Function Template (RFT)
On-the-Fly Analysis using Raster Functions
• You can execute all analysis tools, functions and operators in arcpy• You can perform multidimensional raster algebra using functions and operators
• Subset, RasterCollection, etc. functions allow to manage MD Raster- RasterCollection (rasters, {attribute_dict})
• You can use for each and apply to perform analysis on each slice of a multidimensional raster using any raster function or a function chain
- Apply (in_raster, raster_function, {raster_function_arguments})- Foreach (in_raster, raster_function, {raster_function_arguments})
• You can use arcpy.env.matchMultidimensionalVariable to control analysis
Data Management and Analysis with Multidimensional Raster in ArcPy
• Defines the group of input rasters and attributes to include in a collection, which will be used for processing.
RasterCollection (rasters, {attribute_dict})
Raster Collection
Properties• fields• count
Methods• filter • filterByAttribute• filterByCalendarRange• filterByGeometry• filterByRasterProperty• filterByTime• getFieldValues
• majority • map• max • mean • median • min
• mosaic • qualityMosaic• selectBands• sort • sum • toMultidimensionalRaster
NumPy
• Can be used to extend raster analysis capabilities by creating- custom functions- custom tools
• An extension package to Python- Adds support for large, multi-dimensional arrays- Provides a large library of high-level mathematical functions
• Access the wealth of free functions built by the scientific community - Clustering- Filtering- Linear algebra- Optimization- Fourier transformation- Morphology
NumPy Arrays
• Two functions to work with rasterRasterToNumPyArray(in_raster, {lower_left_corner}, {ncols}, {nrows}, {nodata_to_value})NumPyArrayToRaster (in_array, {lower_left_corner}, {x_cell_size}, {y_cell_size}, {value_to_nodata})
• Raster object Read and Write methods read ({upper_left_corner}, {origin_coordinate}, {ncols}, {nrows}, {nodata_to_value}, {cell_size})
write (array, {upper_left_corner}, {origin_coordinate}, {value_to_nodata})
12 4
343
12 4
343
Numpy Array
Raster
• The PixelBlock object defines a block of pixels within a raster to use for processing. It is used in conjunction with the PixelBlockCollection object to iterate through one or more large rasters for processing.
• getData () method returns a NumPy array within the extentPixelBlock (data, extent)
• The PixelBlockCollection object is an iterator of all PixelBlock objects in a raster or a list of rasters. It can be used to perform customized raster processing on a block-by-block basis, when otherwise the processed rasters would be too large to load into memory.
PixelBlockCollection (in_rasters, pixel_block_size, stride, overlay_type, nodata_to_values)
PixelBlock and PixelBlockCollection
Raster Cell Iterator (RCI)• Visit each cell in a Raster object by row/column
indices and read/write cell values
• Support iterating multiple raster datasets and perform analysis at the same location
• Access to cell values at each location and neighboring locations is efficient
• Available through the Spatial Analyst module
arcpy.sa.RasterCellIterator(*args),
*args is a dictionary that defines rasters and padding in the following format:
{'rasters': [in_rasterobj1, in_rasterobj2, ...], 'padding': padding_factor}
0 1 2 3 4
14 23 32 42 50
14 24 33 43 51
14 23 32 42 51
14 23 32 43 51
14 23 30 39 48
0 1 2 3 4
ArcGIS Platform: Disseminating
Access / Identity
Apps
Services
Desktop Web Device
Online Content and
Services
Server
multivariate multidimensional mosaic dataset
professionalgeospatial analysts
Sharing Scientific Data
• Mosaic Dataset/CRF ≻ Right click from the Catalog and click on ‘Share As Web Layer’
• Map Service (supports WMS)- Makes maps available to the web.
• Image Service (supports WMS)- Provides access to raster data through a web
service.
Enable access to a dynamic representation of your information product as an image service
HRRR Data Sharing Workflow
CreateMosaic Dataset
PublishMosaic Dataset
Ingest:MetadataReferences to Source
Approx 30 records/Sec
ServerAccessible
HRRR Dataset(GRIB)
ArcGISImage Server
DownloadHRRRDataset(GRIB)
ModelGeneration
Time t
High Resolution Rapid Refresh Weather Model3 km resolutionHourly update
Consuming Multidimensional Data Services
• In any ArcGIS application or any WMS client
• In a web map• Identify web services driven by maps or datasets• Bring service layers into a web map
• In a map-based application• Configurable apps• Story Maps• Web AppBuilder• Custom web apps using ArcGIS API for JavaScript
Industry solutions
Workflow apps
App buildersApp templates
Python API Story Maps
Presenting your Actionable Information | Customizing and Extending ArcGIS
Customizing and Extending ArcGIS
Use raster analytics and Python API to monitor coral bleaching in Dashboard to show alert levels and coral stress over time.https://www.esri.com/arcgis-blog/products/image-server/imagery/dev-summit-2020-use-raster-analytics-and-the-python-api-to-monitor-coral-bleaching/
Operations Dashboard for ArcGIS
ArcGIS is a Scientific Collaboration Platform
Con
sum
e
Contribute
Discover
Remote Sensing & Scientific Data
Analysis
Rich Information Products
Dissemination with Choices
FilesServices
CapabilityChoices
FormatChoices
ArcGIS Online or on-premise Portal
Resources
• ArcGIS Pro Help Document- Tutorials in ArcGIS Imagery Workflows - ArcPy classes, functions- ArcPy Image Analyst module- ArcPy Spatial Analyst module- Tool Reference: Multidimension toolbox- Tool Reference: Image Analyst toolbox- Tool Reference: Spatial Analyst toolbox
• Learn ArcGIS Lessons • ArcGIS blogs
Summary
• ArcGIS Platform provides a rich set of functionality to work with multidimensional scientific data
• Multidimensional ribbon, new and enhanced tools makes it easier to visualize, explore and analyze multidimensional raster
• ArcPy Raster object has been enhanced to represent multidimensional raster
• Mosaic Dataset is a robust data model that allows you to manage your large collections of scientific multidimensional data
• Raster function(s) can help with your efficient on the fly computing that saves time and resources
• ArcPy and NumPy is a powerful way to extend analytical capabilities using Python
• Multidimensional data can be shared as a service, it can be consumed in many applications
• ArcGIS is a platform for scientific collaboration