Presented by:
Kevin Sampson, GISP
NCAR WRF-Hydro Development Team
Overview of the ArcGIS WRF-Hydro Pre-processing Tool
CUAHSI Training Workshop | The Community WRF-Hydro Modeling System | May 2-4 2017
Motivation
o Fast, efficient method for producing the ‘routing stack’ and convenience data.
o Consistent processing methodology between domains, regions, datasets.
o Remove the heavy GIS burden from modelers.
To provide WRF-Hydro with a complete set of routing grids that have been hydrologically processed according to user specifications.
Terrain Pre-processing Tools
o WRF-Hydro ArcGIS Pre-processing Toolset
• Fast, efficient method for producing the ‘routing stack’ and other convenience data
• Consistent processing methodology between domains, regions, datasets
WRF-Hydro & ArcGIS
o Desktop GIS Application Suite
o Site-licenses available at most US academic institutions
o Ecosystem of compatible hydrology tools
• Spatial Analyst
• ArcHydro
• TauDEM
o Extensible using Python API (arcpy)
o Handles everything from projections, to analysis, to mapmaking in one library.
o netCDF4 Python included as of 10.3
o https://github.com/NCAR/wrf_hydro_arcgis_preprocessor.git
Requirements
o ArcGIS for Desktop
• Version 10.3.1+ (minimum 10.2.1)
• DO NOT USE ArcGIS 10.4
• Basic, Standard, or Advanced license levels
• Spatial Analyst extension required
• Python 2.7.8, NumPy 1.7.1
• Both installed with ArcGIS Desktop ‘complete’ installation
Python Toolboxes
o Python script wrapped to act as an ArcGIS Toolbox
ToolboxScript (.pyt)
FunctionScript (.py)
o PYT file is the toolbox script containing multiple toolsets
• Functions called from separate script (wrf_hydro_functions.py)
o Parameter handling and validation
WRF-Hydro preprocessor toolbox as viewed from ArcCatalog.
AdvantagesEasy to modifyPortableMany tools organized
Python Toolboxes (.pyt)
o Toolboxes wrapped in Python classes
Pre-ProcessorInputs Outputs
Other parameters
© esri
netCDF file format
o network Common Data Form
• “.nc” extension
o Self describing
• Includes information about the data, coordinate system
o Machine independent
• Usable in many operating systems
o Used in Atmospheric Science
o Multidimensional
• x, y, z, t
y
x
Time
z
FullDom File (Routing Grid)
o netCDF WRF-Hydro input file
o Full High Resolution Domain file (FullDom_hires.nc)
o Stores all routing grids as 2-dimensional variables
o Stores CF-compliant spatial metadata
• grid_mapping
• Projection information
• Coordinate System variable
• ArcGIS-compliant projection information
o Easy to import into GIS Applications
o Ingested by WRF-Hydro
A Note on CF Metadata
o Climate and Forecast Conventions for netCDF data
• Like a standard
• Current version 1.6
• http://cfconventions.org/latest.html
o CF conventions for just about any type data
• Gridded
• Point
• Profile
• timeSeries
o CF-compliant netCDF files make them much easier to use in client applications
• Panoply, ArcGIS, QGIS
Process GEOGRID File
Inputs
o Required:
• WRF GEOGRID file (.nc)
• High-Resolution Elevation
• Elevation file (Esri GRID, GeoTIFF, etc.)
• Mosaic Datasets
o Parameters:
• Regridding Factor – nesting relationship of routing:land grids
• Minimum basin size (in routing grid cells)
• OVROUGHRTFAC - constant
• RETDEPRTFAC – constant
o Optional:
• Station Locations (.csv)
• Lake Polygons (polygon feature class or .shp)
Model Domain
• Area of interest
• Defines model domain
– Size
– Location
– Horizontal resolution
• Defined by geogrid file
– Example:
Input: WRF GEOGRID
o GEOGRID is used in the WRF-Hydro GIS Pre-processor to define the domain’s coordinate reference system, extent, resolution, and certain 2D variables:
• HGT_M (elevation)
• LU_INDEX (landuse)
o Currently supported GEOGRID coordinate systems
• MAP_PROJ = 1 (Lambert Conformal Conic)
• MAP_PROJ = 3 (Mercator)
• MAP_PROJ = 6 (Cylindrical Equidistant but NOT w/ rotated pole)
• MAP_PROJ = 2 (Polar Stereographic)
The purpose of the GEOGRID file is to define the simulation domain and interpolate various static geographical datasets to the model grid.
GEOGRID: Projected Coordinate System
"PROJCS['Lambert_Conformal_Conic',GEOGCS['GCS_Sphere',DATUM['D_Sphere',SPHEROID['Sphere',6370000.0,0.0]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Lambert_Conformal_Conic'],PARAMETER['false_easting',0.0],PARAMETER['false_northing',0.0],PARAMETER['central_meridian',-105.0],PARAMETER['standard_parallel_1',30.0],PARAMETER['standard_parallel_2',50.0],PARAMETER['latitude_of_origin',39.9400138855],UNIT['Meter',1.0]];-36695400 -29251300 10000;-100000 10000;-100000 10000;0.001;0.001;0.001;IsHighPrecision"
esri PE string
Input: Elevation Raster
o Must be an ArcGIS-readable raster format
o Must contain valid coordinate reference system
o Must cover entire extent (and more) of your GEOGRID domain
o Elevation units must be converted to meters (m)
o Should be hydrologically corrected
• Not necessary but helps with channel placement, hydro enforcement, etc.
o Supported datasets include, but are not limited to:
• HydroSHEDS
• NHDPlus (converted from cm to m)
• EU-DEM
Input: Elevation Mosaics
NHD Plus v2 EU-DEM
HydroSheds
Input: Regridding Factor
1000m
100m
GEOGRID Resolution
Regridding Factor= Routing Resolution
1000m
100mRegridding Factor: 10
Raster resolution for terrain processing
1000m 30m
Model Resolution High Resolution
Terrain Pre-processing Workflow
o Resample high-resolution DEM and land use to target resolution and extent using Nearest Neighbor algorithm
o Void-fill the resampled DEM
o D8 Flow Direction
o Flow Accumulation
o Derive CHANNELGRID from flow accumulation raster using threshold of minimum basin size
o Derive Strahler stream order from CHANNELGRID
DEM Resampling Pit-filling Flow directionFlow
accumulationStream
DefinitionStream Order
Basic workflow for terrain pre-processing of WRF-Hydro routing grids.
Reproject & Resample Hydro DEM
o Project input DEM to model projection and domain
o Resample to routing grid resolution
• NEAREST NEIGHBOR resampling
• Finds the ‘nearest neighbor’ pixel in input DEM to each new pixel in the model grid.
• There are more sophisticated methods, but this is fastest and preserves values inside original range of values.
o Re-projection and Resampling can ‘break’ the input HydroDEM.
• Causing artificial ‘pits’.
• Filling in ‘burned in’ areas.
o Even though we start with a HydroDEM, we ‘break’ it, then re-condition it.
Process: Pit Filling
Spatial Analyst “Fill“ Tool
o Fill depressions so that water can roll downhill only. This also creates a smoother DEM than you might find in nature.
o This simple hydro-enforcement method can resolve most issues in a DEM.
o Optional z-limit (global variable) to limit fill depth.
o Problem: some pits are natural, some are not.
© Esri: http://desktop.arcgis.com/en/arcmap/latest/tools/spatial-analyst-toolbox/how-fill-works.htm
Methods: Tarboton et al (1991)
Pit-Filling issues
o …some pits are natural, some are not.
Resampling can break hydrologic connectivity
o Coarsening a HydroDEMcan break hydrologic connectivity.
o Try not to get too much coarser than input HydroDEM, or perform extensive hydro-enforcement on your input DEM first.
o Here, a canyon outlet is filled, causing entire valley to fill during pit filling process.
Flow Direction & Flow Accumulation
o D8 Flow Direction
o Flow Accumulation
1
248
16
32 64 128
Methods: Jenson and Domingue (1988)
© Esri: http://desktop.arcgis.com/en/arcmap/latest/tools/spatial-analyst-toolbox/how-flow-accumulation-works.htm
Stream Definition
o Input Parameter: Number of pixels to define stream
• Yields a minimum ‘basin’ size
• Given in pixels (unitless), on the routing grid
• Affects density of generated channel network
1km2 0.1km2 0.01km2
An analytic method for determining an appropriate threshold value for stream network delineation is presented in Tarboton et al. (1991)
Stream definition
o Use flow accumulation threshold to define channels
o Option: use gaged basins as mask to assign CHANELGRID values
o If reach-based routing is selected, Stream to Feature used to create vector geometry of streams
• streams.shp shapefile written to output directory
Flow Accumulation Channelgrid
Input: Station Location File
o Create in Excel, Numbers, Word, etc.
o Direct output of attribute table from shapefile or feature class
o “LON” & “LAT” required
o If present, basins will be delineated using the points provided
• frxst_basns output variable will be created
• frxst_pts & basn_msk variables will be populated
o If masked to basins, CHANNELGRID will have values -1, 0, -9999
FID,LON,LAT,STATION,Name15,-105.92833,40.08139,Fraser_at_Granby,903330018,-105.9,40.12083,COLO_nr_GRANBY,901950020,-106.3333,39.8803,Blue_R_blw_Grn_Mtn,9057500
Process: Basin Delineation
o Snap points to streams
o ‘Walk’ down channel network a specified distance
• Default = 3 pixel widths (global variable)
o Delineate basin using ‘Watershed’ Spatial Analyst tool
Conejos_nr_Mogote
• Writes output file to: frxst_basns, basn_msk, &gw_basns_geogrid.txt
Basin Masking
o Option to ‘turn off’ channel networks outside gaged basins.
o If gages are provided and the option to mask CHANNELGRID to basins is selected.
• Channel pixels inside gaged basins = 0
• Channels outside = -1
Process: Reach-Based Routing
o CHANNELGRID raster is converted to a shapefile (streams.shp)
o Decomposes line geometry to nodes, and gathers elevation, Latitude and Longitude at each node
o LINKID grid in FullDom file is created and populated with link ID values
o Constructs a .nc file with necessary parameters for reach-based routing:
• Length, Slope, Order, Drop, X/Y, etc.
• Writes output file to Route_Link.nc
Reach-routing Table
Parameter Description
link Link ID (NHDFlowline_Network COMID)
from From Link ID (PlusFlow table FROMCOMID for every TOCOMID)
to To Link ID (PlusFlow table TOCOMID for every FROMCOMID)
lon longitude of the start node
lat latitude of the start node
alt Elevation in meters from the North American Vertical Datum 1988 (NADV88) at start node (MaxElevSmo)
order Stream order (Strahler)
Qi Initial flow in link (CMS)
MusK Muskingum routing time (s)
MusX Muskingum weighting coefficient
Length Stream length (m)
n Manning's roughness
So Slope (meters/meters from NHDFlowline_network.SLOPE)
ChSlp Channel side slope
BtmWdth Bottom width of channel
Kchan Channel conductivity (mm/hr)
x X-coordinate in projected coordinate system
y Y-coordinate in projected coordinate system
o CF-netCDF file containing reach-routing parameters
o Mix of NHD attributes and default values
Process: Stream Order
Stream Order Spatial Analyst tool
• Strahler stream order
• Writes output file to STREAMORDER variable
Route_Link.nc
o netCDF file to store link information.
• 1-Dimension (linkDim)
o CF-netCDF ‘timeSeries’ convention.
DefaultsQi: 0 cmsMusK: 3600 sMusX: 0.2n: 0.035ChSlp: 0.05BtmWdth: 5 mKchan: 0 mm/hr
Process: Latitude & Longitude Grids
o Each pixel on the routing grid will have lat/lon calculated
o Project domain grid to a sphere
o Uses $$XMAP & $$YMAP functions from deprecated arcgisscripting python library.
o Project back to WRF projected coordinate system.
Latitude Longitude
Process: Reservoir Routingo If the option is selected, a polygon shapefile or feature class is
required as input.
o Populates LAKEGRID variable
o Assigns lake ID values to pixels where lakes drain into channel.
o Constructs a LAKEPARM.nc file with necessary variables for reach-based routing:
• Lake area, max elevation, min elevation, base elevation, orifice elevation
LAKEGRID / LAKEPARM.nc
o Input: Reservoirs Shapefile or Feature Class (polygon)
o Size threshold: ‘Threshold’ global variable
• Default: 0.75 km2
o Polygons are resolved on the model grid
o Lake ID (provided) is used to identify lakes
• If no ID is provided, 1-n is used
Original polygons Lakes on the routing grid
Reservoirs and Channels
Lake Routing Table
Parameter Description
Discharge Default Discharge
LkArea Gridded lake area (sq. km)
LkMxH Maximum lake elevation (m ASL)
OrificeA Orifice cross-sectional area (sq. m)
OrificeC Orifice coefficient
OrificeE Orifice elevation (m ASL)
WeirC Weir coefficient
WeirH Weir Height (m ASL)
WeirL Weir length (m)
alt vertical distance above mean sea level in (m ASL)lake_id Lake ID
lat latitude of the lake centroid
lon longitude of the lake centroid
Lakes/Reservoir routing in the NWM
• Lake parameters are stored in the LAKEPARM.nc lake routing table
• Lakes are defined on the 250m routing grid, parameters in the table
Other Grids
o landuse
• GEOGRID LU_INDEX resampled (nearest neighbor) to routing grid
o OVROUGHRTFAC
• Constant 1.0
o RETDEPRTFAC
• Constant 1.0
o LKSATFAC
• Constant 1000.0
GIS Pre-processor Outputs
o Set of netCDF, shapefile, txt (ASCII raster), & log files
• 2-4 netCDF files
• 0-1 ASCII Raster (.txt)
• 0-1 Shapefiles
• 1 .log file
Other Utilities
o ExamineOutputs
• Extracts .zip output file to individual rastersfor viewing in Desktop GIS applications.
o ExportGrid
• Export any M-grid variable from the GEOGRID file to GRID or .TIF format
o ExportPRJ
• Reads GEOGRID file attributes and builds a projection file (.prj)
o GenerateLatLon
• Builds latitude and longitude GRID files from any raster input.
o DomainShapefile
• Creates a polygon shapefile defining the domain boundary.
Customization
o Multiple Toolboxes may be released for different users
• UCAR Internal toolbox takes advantage of available services
• Elevation mosaics and vector services (NHDPlus lakes, flowlines)
• Modular tool structure allows building rapid new toolsets
Tool Messages
Documentation & Test Data
o Detailed documentation
• 40+ page PDF
• Describes tool capabilities, requirements, parameters, and GIS methods used in the tool chain.
o Small GEOGRID domains for testing tool functionality
• Front Range (Lambert Conformal Conic)
• India (Mercator)
o Expected Output provided for comparison
o Required Elevation files (.tif) provided
o Optional stream gages & lakes provided (Front Range)
Bottlenecks / Constraints
o Project High-Resolution Dataset for large areas
• Can be avoided by pre-projecting/resampling high res data
o Flow Accumulation – slowest part of the process
o Not multi-threaded
• Processes runs on one core
• Process not well suited to parallelization
o Windows only GIS platform
THANK [email protected]
References
o Jenson, S. K., and J. O. Domingue. 1988. "Extracting Topographic Structure from Digital Elevation Data for Geographic Information System Analysis." Photogrammetric Engineering and Remote Sensing 54 (11): 1593–1600.
o Tarboton, D. G., R. L. Bras, and I. Rodriguez–Iturbe. 1991. "On the Extraction of Channel Networks from Digital Elevation Data." Hydrological Processes 5: 81–100.
WRF-Hydro GIS Preprocessing ToolsDemo
