http://www.ncl.ucar.edu/Training/Workshops/lectures.shtm

http://www.ncl.ucar.edu/

Slides based on lectures from the NCL workshop

Also see the NCL homepage

http://www.atmos.albany.edu/student/janiga/temp/incoming_grad_lecture.ppt

This ppt can be downloaded off the web at the below address

ncl-talk@ucar.edu

For additional help…

http://www.ncl.ucar.edu/Support/ncl_talk.shtml

[haley@ucar.edu , shea@ucar.edu]

The NCL variable model is based on netCDF

netCDF filesself describing

(ideally) all info contained within the file no external information needed to determine file

contentsportable [machine independent]

Similar to HDF (NASA) and GRIB (NCEP, ECMWF, etc) formats; all these are read by NCL using the same one line.

Supported by many software tools / languages NCL, IDL, Matlab, ferret, GrADS, F, C, C++,Java command line file operators: NCO, CDO

perform various tasks (i.e. concatenation) very efficiently

http://nco.sourceforge.net http://www.mpimet.mpg.de/fileadmin/software/cdo

ncview quick visualization using gui if in COARDS format:

IDV, NCVIEW, Panopoly, etc.

Examining a netCDF file

• ncdump file_name dumps the entire contents of a file; prints every value

• ncdump -h file_name Dumps header information [most commonly used] NCL equivalent: ncl_filedump file_name

• ncdump -v U file_name NCL equivalent: ncl_filedump –v U file_name

• ncl_filedump file_name [ more general ] netCDF3/4, GRIB-1,GRIB-2, HDF, HDF-EOS [HDF5]

• ncview file_name [visualize file contents]

Now for an example….

Parts of netCDF file

DIMENSION SIZES dimensions: lat = 64 lon = 128 time = 12

VARIABLES: Names , Types, Attributes, Coordinate Variablesvariables: float lat(lat) lat:long_name = "latitude" lat:units = "degrees_north" float lon(lon) lon:long_name = "longitude" lon:units = "degrees_east" int time(time) time:long_name = "time" time:units = "Month of Year" double T(time, lat, lon) T:long_name = “Temperature” T:units = “C" T:missing_value = 1.e+20f T:_FillValue = 1.e+20f

FILE ATTRIBUTESglobal attributes: title = “Temp: 1999” source = “NCAR” conventions = "None”

Longitude coordinate variable (1D, &)

Latitude coordinate variable (1D, &

)

attributes @:

• long_name

• units

visual: simple 2D netCDF Variable

coordinate variables (rectilinear grid)

Non-Traditional Grids or Arrays• netCDF and NCL can handle various “other” types of data:

Spectral coefficents (YOTC) Lambert conformal (WRF)

http://www.ncl.ucar.edu/Applications/wrflc.shtml Radiosondes Surface station data Hurricane tracks

http://www.ncl.ucar.edu/Applications/unique.shtml Make your own!

• Tons of pre-defined functions for manipulating and converting between different formats as well as plotting.http://www.ncl.ucar.edu/Document/Functions/list_alpha.shtm

netCDF [NCL] Variable model

f = addfile(“foo.nc”,”r”) ; grb/hdfx = f->X

XScalar

or Array

attributeslong_name_FillValue

unitsadd_offset

scale_factoretc. values

Scalar or

Array

attributeslong_name_FillValue

unitsadd_offset

scale_factoretc.

accessed via @ accessed via &

timelevlatlonetc.

coordinates timelevlatlonetc.

coord var

NCL reads the scalar/array, attributes, and coordinate

variables as an object

X

Detailed Look netCDF Variable (NCL)ncl <return> ; interactive mode

ncl 0 > f = addfile ("UV300.nc", "r") ; open file

ncl 1 > u = f->U ; import STRUCTURE

ncl 2 > printVarSummary (u) ; overview of variable

Variable: uType: floatTotal Size: 65536 bytes 16384 valuesNumber of Dimensions: 3Dimensions and Sizes: [time | 2] x [lat | 64] x [lon | 128]Coordinates: time: [ 1 .. 7 ] lat: [ -87.8638 .. 87.8638 ] lon: [ 0 .. 357.185]Number of Attributes: 5 _FillValue : 1e36 units : m/s long_name : Zonal Wind Component short_name : U missing_value : 1e36

Classic netCDFVariable Model

NCL syntax/funcs query use modify add any aspect of data object

NCAR Command Language• Complete Programming Language

data types variables operators expressions conditional statements loops functions/procedures/graphics

• Features query / manipulate meta data import data in a variety of formats array syntax / operations can use user fortran/C codes and commercial libraries most functions/procedures ignore missing data

Running NCL• Interactive Mode (Command line)

ncl [options][command-line-arguments] <return> ncl> enter commands ncl> quit <return>

can save interactive commandsncl> record “file_name” ncl> stop record

• Batch Mode [ .ncl suffix is optional] ncl [options][arguments] script.ncl

ncl < script.ncl [also acceptable] nohup ncl [options][arguments] script.ncl > out.txt &"&" means put in background“nohup” means don’t quit when I logoff (ie long scripts).

NCL predefined options• ncl –hnxV [predfined options are preceded by dash]• may be used for interactive or batch mode • informational

ncl –h [display predefined options and usage and exit] ncl –V [print the NCL version and exit]

• action ncl –x [echo statements as encountered (debug)] ncl –n [don't enumerate dimensions of values in print() ]

• multiple predefined options ncl –nx [ not ncl –n –x ]

http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/NclCLO.shtml

Variables can be set on the command line prior to a script. However, this tends to not be necessary.Input from a text file or systemfunc()

Sample Batch Script: sample.ncl load “$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl“load “$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl“

f1 = addfile("TMP_58-97.nc", "r") ; open netCDF file T = f1->Tmp ; T(time,lev,lat,lon) =>(480,17,73,144) f2 = addfile(“P_1958-1997.grb", "r") ; open GRIB file P = f2->Pres ; P(time,lev,lat,lon) f3 = addfile(“Q_1958-1997.hdfeos", "r") ; open hdf –eos file

Q = f3->Specific_Humidity ; Q(time,lev,lat,lon)

wks = gsn_open_wks("ps", “sample") ; open a graphic workstation gsn_define_colormap (wks,"gui_default") ; change from default

plot = gsn_csm_contour_map_polar (wks,T(0,5,:,:),False) res = True ; change default plot res@cnFillOn = True ; use colors res@gsnSpreadColors = True ; use entire color map plot = gsn_csm_pres_hgt (wks,T(0,:,{50},:), res )

pot = T* (1.+0.622*Q) *(1000/P)^0.286 ; potential temperature (array)

print("pot: min="+ min(pot) + " max="+ max(pot) )

begin ; optional

end ; only if begin is present

Outline: NCL Language Basics

• special syntax characters• data types• Variables netCDF/NCL variable model • attributes • _FillValue• named dimensions• coordinate variables• print and printVarSummary• shaping • subscripting

NCL Syntax Characters• ; - comment [can appear anywhere]• @ - reference/create attributes• ! - reference/create named dimension• & - reference/create coordinate variable•{…} - coordinate subscripting• $ - enclose strings when (im/ex)port variables via addfile•(/../) - array construct characters• : - array syntax• | - separator for named dimensions• \ - continue character [statement to span multiple lines]• :: - syntax for external shared objects (eg, fortran/C)• -> - use to (im/ex)port variables via addfile function•“ “ - use for a string

Data TypesNumeric• double (64 bit)• float (32 bit)• long (32 or 64 bit)• integer (32 bit)• short (16 bit)• byte ( 8 bit, 0-255)• complex NOT supported

non-Numeric• string• character• graphic• file• logical• list

Time coordinate variables frequently double.

If IR data (K) was offset by 100 it could be stored in byte instead of double (8 times less disk space) or float (4 times less disk space).

Data will often be packed into short or byte this way xFloat = short2flt(xShort)xShort@scale_factorxShort@add_offset

Attributes

• assign/access with @ character T@units = “deg C” T@long_name = “temperature” T@whatever = an array of any dimension or type T@_FillValue = -999. plot title uses x@long_name by default

• attribute functions [isatt, getfilevaratts] if (T@units.EQ.”K”) then

• delete can eliminate an attribute delete(T@title)

• info about a variable or file [meta data] attributes can be any data type except file scalar, multi dimensional array (string, numeric)

_FillValue• Unidata and NCL reserved attribute • most NCL functions ignore _FillValue

• netCDF Operators [NCO]: _FillValue attribute• ncview: missing_value attribute

best to create netCDF files with both

• NCL: don’t use commonly encountered values as a _FillValue!

• Note: “missing_value” attribute: no special status to NCL if “T” has “missing_value” attribute but no “_FillValue ”

NCL creates: T@_FillValue = T@missing_value use built-in function “ismissing” to check for _FillValue

if (any(ismissing(T))) then T = where (ismissing(T),0,T)

Arrays

• row major ….. like C/C++ left dimension varies slowest; right varies fastest dimension numbering left to right [0,1,..]

• indicies [subscripts] are zero based [0,N-1]

Consider T(:,:,:) left dimension is 0 [ T!0 ] middle dimension is 1 [ T!1 ] right dimension is 2 [ T!2 ]

NCL (netCDF) Dimensions

• assigned with ! character {let T(:,:,:)} T!0 = "time" ; leftmost [slowest varying] dim T!1 = "lat“ T!2 = "lon" ; rightmost [fastest varying] dim

• dim names may be renamed, retrieved T!1 = "LAT" … dName = T!2

• delete can eliminate: delete (T!2)

• Named dimensions used to reshape

• may be “named”provides alternative way to reference subscriptsrecommendation: always name dimensionsuse NCL syntax

NCL (netCDF) Coordinate Variables• data coordinates [ eg: time, level, lat, lon ]

strict netCDF definition 1D numeric array monotonically in[de]creasing numeric values can only be assigned to a named dimension 1D array must be the same size as dimension should not have _FillValue attribute

coordinate values assigned via & character T&lat = latitude ; Here latitude is a 1D array

• used in graphics, coordinate subscripting

• coordinate functions iscoord , isfilevarcoord

• delete can eliminate coordinate arraydelete(T&time)

Create and Assign Coordinate Variables

• assign values to named dimension time&time = time lon&lon = lon

• let x be 2D: name dimensions x!0 = “time” … x!1 = “lon”

• assign coordinate variables to x x&time = time … x&lon = lon

• assign dimension name [same as variable name] time!0 = “time” lon!0 = “lon”

• create 1D array time = (/ 1980, 1981, 1982 /) ; integer lon = (/ -90, ... 90 /) ; float

Access/Change/Create/Delete Meta Data• @ attributes

u@long_name = "U" lonName = u@long_name

• ! named dimensions u!0 = "TIME" tName = u!0

• & coordinate variable u&lat = (/ -90., -85, .... , 85., 90. /) latitude = u&lat

• $ substitute string x = fin->$variable(n)$(date,{-30:60},:)

Variable Assignment

• Value-only assignment (no meta copy) U multi-dimensional array with meta data

Uval = (/ U /) or Uval = (/ f->U/) the (/ ... /) operator pair strips meta data

• Variable-to-Variable assignment consider y = x where x is previously defined

if y not defined: y has same type/shape/values/meta data as x

if y predefined: y must have same shape and type or, x must be coerceible to the type of y y attributes, dimension names and coordinate variables, if they exist, will be over written

Variable Subscripting (1 of 3)

Standard Array Subscripting• ranges: start/end and [optional] stride• indices separated by :• omitting start/end index implies default begin/end

Consider T(time,lat,lon)T entire array [ don't use T(:,:,:) ]T(0,:,::5) 1st time index, all lat, every 5th lon T(0,::-1,:50) 1st time index, reverse lat order, 1st

51 lon T(:1,45,10:20) 1st 2 time indices, 46th index of lat,

10-20 indicies of lon

Variable Subscripting (2 of 3)

Coordinate Variable Subscripting• only applies to coordinate variables• same rules apply for ranges, strides, defaults• use curly brackets {…}• standard and coordinate subs can be mixed [if no reorder]

T(:,{-30:30},:) all times/lon, lat -30° to +30° (inclusive)

T(0,{-20},{-180:35:3}) 1st time, lat nearest - 20°, every 3rd lon between -180° and 35°

Variable Subscripting (3 of 3)

Named Dimensions• only used for dimension reordering• indicated by |• dim names must be used for each subscript• named/coordinate subscripting can be mixed

Consider T(time,lat,lon)

t = T(lat|:, lon|:, time|:) makes t(lat,lon,time)

t = T(time|:,{lon|90:120},{lat|-20:20}) all times,

90-120° lon, -20-20° lat

Longitude coordinate variable (1D)

Latitude coordinate variable (1D)

Standard and Coordinate Subscripting

Standard:

T(9:13,1:8)

Coordinate:

T({-10:20},{-170:-110})

Combined:

T({-10:20}, 1:8)

“printing”• printVarSummary

gives gross overview of a variable

• print same info as printVarSummary prints values

• write_matrix print to standard out or a file format control of numerical output can write to file also

print (1 of 2)

Variable: TType: floatTotal Size: 32768 bytes 8192 valuesNumber of Dimensions: 2 Dimensions / Sizes: [lat | 64] x [lon | 128]Coordinates:

lat: [-87.86379 .. 87.86379] lon: [ 0. 0 .. 357.1875]Number of Attributes: 2 long_name: Temperature units: C(0,0) -31.7(0,1) -31.4(0,2) -32.3(0,3) -33.4(0,4) -31.3 etc. [entire T array will be printed]

• Prints out all variable information including meta data, values T(lat,lon): print (T)

print (2 of 2)

Variable: T (subsection)Type: floatTotal Size: 256 bytes 64 valuesNumber of Dimensions: 1 Dimensions / Sizes: [lat | 64] Coordinates:

lat: [-87.86379 .. 87.86379]Number of Attributes: 3 long_name: Temperature units: C lon: 109.6875 [ added ](0) -40.7(1) -33.0(2) -25.1 (3) -20.0(4) -15.3 etc.

• can be used to print a subset of array meta data, values T(lat,lon): print( T(:,103) ) or print( T(:,{110}) )

Things that can be done (look up or ask)

• Write netCDF files (compression, making large files)• Read and write binary and ascii• Functions and procedures in NCL language (over 1000),

fortran or C (most are earth science orientated; essentially Matlab or IDV for earth science).

• Loops, conditional statements.• OPeNDAP support (query and read files directly from the

web without downloading the whole file)

• http://www.ncl.ucar.edu/Document/Functions/index.shtml