How to Regrid Swath Data

Charlie Zender <zender@uci.edu>

Departments of Earth System Science and Computer Science, UC Irvine

Seminar on Web

GSFC GES DISCGreenbelt, MD April 5, 2016

Weight f(x,y) numbers, then add them

Regridding Swath, Curvilinear, Rectangular, and Unstructured

Data (SCRUD)Charlie Zender <zender@uci.edu>

Departments of Earth System Science and Computer Science, UC Irvine

Seminar on Web

GSFC DISCGreenbelt, MD April 5, 2016

Overview1.Prerequisites2.Rectangular, Swath/Curvilinear, Unstructured Grids3.Remapping4.Parallelism and Performance

Prerequisites1. netCDF Operators (NCO) (mandatory)> sudo dnf install nco

2. ESMF_RegridWeightGen (optional)> sudo dnf install gfortran csh ncl

3. TempestRemap (optional)> sudo dnf install lapack lapack-devel blas> git clone https://github.com \ ClimateGlobalChange/tempestremap.git

Find/Obtain/Install NCO DOE Labs, NCAR Yellowstone:> export PATH='~zender/bin':${PATH}> export LD_LIBRARY_PATH='~zender/lib': \${LD_LIBRARY_PATH}

Local clusters, workstations, laptops:> sudo aptitude install nco # Debian> sudo dnf install nco # Fedora> sudo port install nco # MacPorts> git clone git@github.com:nco/nco.git # DIY> ./configure; sudo make install

GitHub NCO homepage

Overview1.Prerequisites2.Rectangular, Swath/Curvilinear, Unstructured Grids3.Remapping4.Parallelism and Performance

RectangularEquiangular Grid

RectangularCap Grid

Courtesy M. Taylor, Sandia Nat'l Lab

Curvilinearlat(x,y), lon(x,y)

UnstructuredCubed-Sphere lat(i), lon(i)

Courtesy M. Taylor, Sandia Nat'l Lab

UnstructuredDrainage/river lat(i), lon(i)

SwathCurvilinearlat(x,y), lon(x,y)

Potentially SwathUnstructuredlat(i), lon(j)

Overview1.Prerequisites2.Rectangular, Swath/Curvilinear, Unstructured Grids3.Remapping4.Parallelism and Performance

Regrid NASA AIRS Level 2 Swath Data in raw HDF4 format from regional curvilinear 45x30 source grid to equiangular 1x1 degree:

Remap Swath Data

% ncremap -i AIRS.2014.10.01.202.L2.RetStd.v6.0.11.0.G14275134307.hdf -d 1x1.nc

Global Sea-Ice (CICE) Data

Native CICE grid has _FillValue in mask variable (tmask), which NCL function curvilinear_to_SCRIP() and ESMF do not understand. Unless user first manually sets _FillValue to 0, NCL generates incorrect grid, and regridding produces subtly biased results. ncremap handles missing values in masks without user intervention.

"ncremap" documentation

Run "ncremap"> ncremap -i in.nc -d dst.nc -o out.nc> ncremap # Print help

ncremap Help Screen

Examples

Command

Options

Pre-computed map > ncremap -I in_drc -m map.nc -O out_drc

Data on Identical input grids

Standard Output> ls in*.nc | ncremap -m map -O drc

Summary

Status

Preamble

Regrid NASA AIRS Level 2 Swath Data in raw HDF4 format from regional curvilinear 45x30 source grid to equiangular 1x1 degree:

Remap Swath Data

% ncremap -i AIRS.2014.10.01.202.L2.RetStd.v6.0.11.0.G14275134307.hdf -d 1x1.nc

Four Invocation ModesFree-will: Infer source and destination grids> ncremap -i in.nc -d dst.nc # L2-Swath Data!

Old grid: Known grid(s) generate mapfile> ncremap -s src.nc -g dst_grd.nc

New grid: Generate destination-grid with NCO> ncremap -G '--rgr latlon=40,40 --rgr \ snwe=30.0,70.0,-130.0,-90.0'

Pre-Destination: Apply supplied mapfile> ncremap -m map.nc # Fastest

Free-will Grid-inferral Algorithm

ncremap first looks in any (optionally) supplied gridfile or mapfile, and supplements this information (if any) with grid details sometimes provided in CF metadata (e.g., "bounds" variables) from the datafiles themselves. ncremap next tests for known rectangular grid types (equiangular, FV, offset, Gaussian) and supplements metadata with exact information derived from inferred gridtype, if any. This fails for swath, curvilinear, and unstructured grids, for which ncremap must extrapolate grid properties from cell-center locations.

Regrid NASA MODIS Level 2 Data in HDF4 format from 550x543 swath to equiangular 1x1 degree:

Remap Swath Data

% ncremap -v chlorophyll -i MODIS_L2N_20140304T1120_noNaN. -d 1x1.nc

Regrid NASA OMI Level 2 Ozone Data in HDF-EOS format from swath to 1x1 degree:

Remap Swath Data

% ncremap -v StepTwoO3 -i OMI-Aura_L2-OMTO3_2015m0731t0034-o58727_v003-2015m0731t080836.he5.nc -d 1x1.nc

Regrid NASA OMI Level 2 Ozone Data in HDF-EOS format from swath to 1x1 degree:

Remap Swath Data

% ncremap -v StepTwoO3 -i OMI-Aura_L2-OMTO3_2015m0731t0034-o58727_v003-2015m0731t080836.he5.nc -d 1x1.nc

ncremap Options> ncremap -i in.nc -d dst.nc -o out.nc> ncremap … -m map … # Map (weights)> ncremap … -I drc_in … # Input directory> ncremap … -O drc_out … # Output directory> ncremap … -v FSNT,Q … # Subset> ncremap … -a conserve … # Algorithm> ncremap … -j 16 … # Batch job number> ncremap … -P airs … # Permutation type> ncremap … -p mpi … # Parallelism type> ncremap … -t thr_nbr … # Thread number> ncremap … -w tempest … # Weight generator

Current Issues 1. Regridding unstructured data when horizontal dimensions (i.e., latitude/longitude) vary more slowly than others (e.g., vertical, species)> ncremap … -P airs … # AIRS, HIRDLS, MLS (beta)2. Extensive variables (e.g., numbers, counts)> ncremap … -x TsurfStd_ct … # In progress (alpha)3. LIDAR without cell bounds or areas> ncremap … -i oco2.nc4 ... # In progress (alpha)4. Overlapping gridcells, e.g., polar swaths> ncremap … -i gpm.nc, amsr.nc ... # In progress

Overview1.Prerequisites2.Rectangular, Swath/Curvilinear, Unstructured Grids3.Remapping4.Parallelism and Performance

Three Parallelism Modes> ncremap -I drc_in -O drc_out> ncremap … -p nil … # Serial> ncremap … -p bck ... # Background (default)> ncremap … -p mpi … # MPI

● Serial: Execute sequentially on single node● Background: Fork process for each remap all on

single node. Control batchsize with -j job_nbr.● MPI: Spawn MPI process to new node for each

remap. Control batchsize with -j job_nbr.

swath1

core1 core2 … coreC

●1 remap per core, C cores, thr_nbr=1●1 remap per C cores, thr_nbr=C●Hybrid

swathNswath2

Background Parallelism &

swath1

node1 node2 … nodeN

●1 remap per node, N nodes, thr_nbr=C●2 remaps per node, thr_nbr=C/2

swathNswath2

MPI Parallelism

Threaded Regridding

●Regridding automatically invokes 2 (default) to 8 threads

●> ncremap…-t 8... # Threads

node1

var1, var4, ... varV-2

core1

core2

core3

var2, var5, ... varV-1

var3, var6, ... varV

ncremap is threaded over variables with OpenMP and scales well up to 8-16 threads:

Threading Performance

Three regridders (UV-CDAT, NCL, NCO) on CAM-SE unstructured grid datasets from ~1-13 GB on multicore nodes:

Regridder Bake-Off

node1core1

core2

coreC

var2, var5, ... varV-1

var3, var6, ... varV

node2 nodeN

core2

core1 core1

core2

coreC coreC

var1, var4, ... varV-2

var1, var4, ... varV-2

var2, var5, ... varV-1

var2, var5, ... varV-1

var3, var6, ... varV

var3, var6, ... varV

var1, var4, ... varV-2

Chooose Parallel Mode to Optimize Throughput

RAM: sizeof(map) + ~4x sizeof(var) per thread● Serial: Safest, slowest. Ideal for laptops, desktops.

Alternative to MPI-mode for small RAM systems.● Background: Default mode. Works on login nodes,

compute nodes, and beefy workstations.● MPI: Necessary for fast high-resolution remapping

on small-moderate systems

Vaporware 1. Allow CF "coordinates" to specify coordinates2. Radius-of-influence for weighting3. Understand quality flags4. Optimal Voronoi input mesh

ncremap Advances1. Single command regrids Swath, Curvilinear,

Rectangular, and Unstructured Data (SCRUD)2. Infers accurate grids without user intervention3. Fastest regridder tested (with pre-computed maps)4. Parallelizes remapping workflows

Supplementary Slides