Computers & Geosciences 33 (2007) 62–75 The mgd77 supplement to the generic mapping tools $ Paul Wessel , Michael T. Chandler Department of Geology and Geophysics, School of Ocean and Earth Science and Technology, University of Hawaii at Manoa, 1680 East-West Rd., Honolulu, HI 96822, USA Received 17 March 2006; received in revised form 17 May 2006; accepted 26 May 2006 Abstract Marine geophysical data collected by government and academic vessels are archived at the US National Geophysical Data Center in Boulder, Colorado. Data exchanges between NGDC and source institutions use an ASCII, punch-card format known as the MGD77 format, reflecting a style of file design common in the 1970s. We have developed a set of new software tools that can convert between this exchange format and a new COARDS-compliant, netCDF-based, architecture-independent file format that we call the MGD77+ format. The new mgd77 tools allow the data to be manipulated in a variety of ways useful for marine research. These tools are written in POSIX-compliant C, are distributed as a supplement to the Generic Mapping Tools, and can be installed on any computer platform. Because the new format is COARDS and CF-1.0 compliant, the files can be read by any general-purpose program capable of decoding these standards; a welcome side effect. One such program is the Java application ncBrowse developed by NOAA. Furthermore, the more compact netCDF files have file sizes that are, on average, only 30% of the original sizes. Because procedural changes at NGDC and source institutions necessarily occur infrequently, it is expected that the MGD77 format will remain the official exchange format for underway geophysical data for some time, whereas the new MGD77+ format offers users much needed flexibility in how they use the data. r 2006 Elsevier Ltd. All rights reserved. Keywords: MGD77; GMT; Marine geophysics; netCDF; NGDC 1. Introduction Underway geophysical data collected by both US and foreign research vessels are stored and dis- tributed by the US National Geophysical Data Center (NGDC) in Boulder, Colorado. These data are fundamental to marine geophysical research: they supplied the observational basis that enabled seminal contributions to Earth science during the plate tectonic revolution (e.g., Heezen, 1960; Raff and Mason, 1961; Pitman and Heirtzler, 1966) and they continue to play a critical role in new discoveries and serve as ‘‘ground truth’’ for the calibration of satellite-derived global grids (Sand- well and Smith, 1997; Smith and Sandwell, 1997). While submitting data to NGDC is only a require- ment for NSF-funded cruises, most international oceanographic institutions collecting unclassified data contribute their data to NGDC as well. This arrangement has served the community of marine researchers well: more than five decades of data, ARTICLE IN PRESS www.elsevier.com/locate/cageo 0098-3004/$ - see front matter r 2006 Elsevier Ltd. All rights reserved. doi:10.1016/j.cageo.2006.05.006 $ Code available from server at http://gmt.soest.hawaii.edu. Corresponding author. Tel.: +1 808 956 4778; fax: +1 808 956 5154. E-mail address: [email protected] (P. Wessel).
Transcript
ARTICLE IN PRESS
0098-3004/$ - se
doi:10.1016/j.ca
$Code avail�Correspond
fax: +1808 956
E-mail addr
Computers & Geosciences 33 (2007) 62–75
www.elsevier.com/locate/cageo
The mgd77 supplement to the generic mapping tools$
Paul Wessel�, Michael T. Chandler
Department of Geology and Geophysics, School of Ocean and Earth Science and Technology, University of Hawaii at Manoa,
1680 East-West Rd., Honolulu, HI 96822, USA
Received 17 March 2006; received in revised form 17 May 2006; accepted 26 May 2006
Abstract
Marine geophysical data collected by government and academic vessels are archived at the US National Geophysical
Data Center in Boulder, Colorado. Data exchanges between NGDC and source institutions use an ASCII, punch-card
format known as the MGD77 format, reflecting a style of file design common in the 1970s. We have developed a set of new
software tools that can convert between this exchange format and a new COARDS-compliant, netCDF-based,
architecture-independent file format that we call the MGD77+ format. The new mgd77 tools allow the data to be
manipulated in a variety of ways useful for marine research. These tools are written in POSIX-compliant C, are distributed
as a supplement to the Generic Mapping Tools, and can be installed on any computer platform. Because the new format is
COARDS and CF-1.0 compliant, the files can be read by any general-purpose program capable of decoding these
standards; a welcome side effect. One such program is the Java application ncBrowse developed by NOAA. Furthermore,
the more compact netCDF files have file sizes that are, on average, only 30% of the original sizes. Because procedural
changes at NGDC and source institutions necessarily occur infrequently, it is expected that the MGD77 format will remain
the official exchange format for underway geophysical data for some time, whereas the new MGD77+ format offers users
Underway geophysical data collected by both USand foreign research vessels are stored and dis-tributed by the US National Geophysical DataCenter (NGDC) in Boulder, Colorado. These dataare fundamental to marine geophysical research:they supplied the observational basis that enabled
e front matter r 2006 Elsevier Ltd. All rights reserved
seminal contributions to Earth science during theplate tectonic revolution (e.g., Heezen, 1960; Raffand Mason, 1961; Pitman and Heirtzler, 1966) andthey continue to play a critical role in newdiscoveries and serve as ‘‘ground truth’’ for thecalibration of satellite-derived global grids (Sand-well and Smith, 1997; Smith and Sandwell, 1997).While submitting data to NGDC is only a require-ment for NSF-funded cruises, most internationaloceanographic institutions collecting unclassifieddata contribute their data to NGDC as well. Thisarrangement has served the community of marineresearchers well: more than five decades of data,
exceeding 64 million individual records (Fig. 1),have been archived and are available from NGDCvia DVD-ROM and anonymous ftp. Data ex-changes between contributing agencies, NGDC,and the end users are done using the MGD77format, an ASCII, Fortran-style punch card formatthat originated by consensus in 1977 (Hittelmanet al., 1977). In the late 1980s, a binary file formatcalled the ‘‘.gmt’’ format was introduced as alightweight version containing only navigationinformation (lat, lon, time) and the three data fieldsfree-air anomaly (g), magnetic anomaly (m), andcorrected depth (t) (Wessel and Smith, 1991). Byexcluding the bulk of the MGD77 data record, thenew format resulted in much smaller files, and beingbinary they could be read very rapidly. The currentGeneric Mapping Tools (GMT) mgg supplementsupports this legacy format (Wessel and Smith,1991; Wessel and Smith, 1995; Wessel and Smith,1998). However, the exclusion of original observa-tions such as two-way travel-times and magnetictotal field values and all data codes and processingflags would limit this format’s usefulness. In theearly 1990s, scientists at Scripps (Sandwell, Cande,and Smith) extended the .gmt format to carryseveral optional data columns such as filteredmagnetics and satellite gravity sampled along track;the resulting data files were supported by a clonedgmt+ package derived from the original mgg
Fig. 1. MGD77 database of marine geophysical track-line data. Shown
totaling almost 43 million records. Mollweide equal-area projection.
supplement (Sandwell, 2006, pers. comm.). Re-cently, we began the development of a new mgd77supplement to GMT, capable of reading and writingthe original ASCII MGD77 files as well assupporting a modern, netCDF-based format. Inaddition, the new tools were designed to extractnamed columns in any order, select output recordssubject to flexible, logical tests, manage the applica-tion of systematic corrections to faulty data, andplot the ship track on GMT maps. The newsupplement greatly extends the functionality of theolder mgg supplement. In addition, work was alsostarted on x2sys, the replacement for the olderx_system crossover suite (Wessel, 1989), which canonly read the old ‘‘.gmt’’ format, with intention tomake it file-format agnostic. Some of the databasebook-keeping in mgg have now been generalized toany data file format and placed in the x2sys suite;the latter is still under redesign and development.
Like most large-scale databases assembled overseveral decades, the NGDC data contain significanterrors. Some of these have been documentedpreviously (Wessel and Watts, 1988; Smith, 1993),and new and comprehensive clean-up efforts arewell underway (Chandler and Wessel, 2004; Beckerand Sandwell, 2006). This effort considers all thedata in a MGD77 file, not just gravity andbathymetry. Because NGDC acts as a library ofinformation, error corrections must be implemented
are locations of all individual depth soundings archived at NGDC,
as errata tables that can be used to make correctionswhen data are extracted. Part of the project involvesidentifying data problems and providing meta-datainformation that will allow users to either makecorrections on the fly (say, convert wrongly scaledvalues to a correct unit) or skip unsalvageable data,leaving the original, archived MGD77 data setintact. Many geophysical applications, such asbathymetric prediction, require original (i.e., notinterpolated) soundings for calibration (Smith andSandwell, 1997; Calmant et al., 2002), and valida-tion of altimetric gravity requires comparisons withshipboard gravity measurements (e.g., Marks,1996). Finally, custom-designed bandpass-filteredmagnetic anomalies are often used to identifyisochrones (e.g., LaBrecque et al., 1985). It is thususeful to allow for the inclusion of extra datacolumns–reflecting the original purpose of Scripps’gmt+ system. However, for the data clean-up taskto be successful and portable to NGDC (i.e., so theresults become available to users via NGDC) it isessential that all the information in the file is keptintact. The original reasons for strongly decimatingthe MGD77 files—size and speed of i/o—have lostmuch of their relevance. Size requirements aremoderate (NGDC ASCII data occupy o7.5Gb),and by choosing a binary format we eliminate theASCII-to-binary conversion bottleneck.
The mgd77 supplement is an undertaking de-signed to: (1) improve on the limited functionality ofthe existing mgg supplement, (2) incorporate someof the ideas from Scripps’ gmt+ modification byallowing extra data columns to be added to the files,(3) add new capabilities for managing marinegeophysical trackline data stored in an architec-ture-independent CF-1.0- and COARDS-compliantnetCDF file format, and (4) remain fully compatiblewith the official MGD77 exchange format. Thepurpose of this manuscript is to give an overview ofthe mgd77 supplement, present some of the under-lying ideas, and to illustrate the maintenance of acollection of NGDC-supplied MGD77 files usingthese new tools. Being released as a supplement toGMT means the mgd77 supplement will receiveproper updates as GMT continues to evolve. LikeGMT, the supplement is written in POSIX C andhas been tested on both UNIX (including Linux andMac OS X) and Windows. Users developing theirown applications that need to handle MGD77+data efficiently (such as GUI editors) may link withthe mgd77 API library and call the availablefunctions. To install mgd77, simply install GMT
and make sure the mgd77 supplement has beenincluded; for more information, see http://gmt.soest.hawaii.edu. Furthermore, a separatemgd77 sub-page under the gmt site provides themost up-to-date information about the supplement.
2. The MGD77 and MGD77+ file formats
The mgd77 tools, which we will describe inSection 3 below, can work with three different fileformats: the original MGD77 format (file extension‘‘.mgd77’’) of files maintained by NGDC, our newnetCDF, CF-1.0- and COARDS-compliant format(extension ‘‘.nc’’), and a plain ASCII table with tab-separated columns suitable for manual editing(extension ‘‘.dat’’). A brief description of eachformat follows; Table 1 lists the various variablesthat are stored in all of these files and their storedprecision. The formats are:
1.
MGD77: A proper MGD77 file from NGDCcontains a header section (24 records of 80 byteseach) and any number of data records (120 byteseach). Each data record holds 24 differentfloating-point values (navigation data such aslatitude, longitude, and time, geophysical obser-vations such as two-way travel times, gravity,and magnetics, a mix of correction terms such asEotvos correction and diurnal variation, andvarious processing codes) as well as three strings(survey ID, seismic line number, and seismicshot-point number). All decimal points areimplied, thus all values are essentially written asintegers. Missing values are flagged, typicallywith strings of 9s. For cruises with only somegeophysical values, much of the data file consistsof strings of 9s.
2.
MGD77+: We have designed a new file format,which we call MGD77+. It contains all theinformation present in the original MGD77 files,but as a proper COARDS-compliant netCDF fileit also holds several global attributes (file author,history, title, file format version, and convention)as well as attributes for individual variables (suchas variable name, range, missing value flags, scalefactors, and more). It is the presence of suchattributes that allows netCDF files to be self-documenting (Rew and Davis, 1990).
3.
DAT: Finally, we also support a plain ASCIItable consisting of tab-separated columns. Itserves the purpose of being a simple format thatis easy to read and edit in a text editor (the lack
of column separators in the MGD77 formatmake manual editing problematic). The headersof the MGD77 files are simply replicated in thesetables.
As we will see, the mgd77 programs have optionsto select which format to read and write.
3. The mgd77 family of tools
There are seven tools in the mgd77 suite of tools,all with different purposes. We will discuss thembriefly and give an overview of each tool’s capabilityand purpose. In the final section we will cover theinitial installation and setup of a MGD77+database based on NGDC data. However, we willleave the detailed discussion of the meaning of eachprogram option to the documentation pages, whichare supplied in UNIX man, PDF, and HTML
formats. Table 2 gives a brief listing of the tools andtheir main functions. Note that we describe thesetools with data managers and data-contributingscientists as the intended audience, i.e., thoseresponsible for putting together new data files,assessing errors, and applying corrections. Theordinary users of NGDC data, having perhapsread-only access to the data bases, are likely tofocus their attention on the use of the dataextracting (mgd77list) and track-line plotting(mgd77track) tools; however, all users need to beaware of the underlying assumptions and correc-tions implemented by their data managers for theoptimal use of NGDC data.
3.1. Converting between file formats
The tool mgd77convert serves the purpose ofconverting between the three representations of the
ARTICLE IN PRESS
Table 2
Components of mgd77 supplement
Tool Purpose
mgd77convert Convert between three supported file formats
mgd77info Display metadata information about specified cruises
mgd77list Extract data records from specified cruises
mgd77manage Augment files with additional data columns
mgd77path Return system path to specified cruises
mgd77sniffer Perform along-track quality control for specified cruises
mgd77track Plot tracks of specified cruises on a GMT map
MGD77 data: the original MGD77 format, our newnetCDF, CF-1.0- and COARDS-compliant format,and a plain ASCII tab-separated table suitable formanual editing. The conversions are 100% rever-sible; no data or metadata are lost in the transla-tions. The synopsis for mgd77convert is (optionalarguments are listed in brackets):
mgd77convert cruises
� Fajcjt�T½þ�ajcjt½�L½e�½w�½þ�� ½�V�½�4�
where the –F and –T options specify the ‘‘from’’ and‘‘to’’ formats. Use –T+ to overwrite existing files.The other options select the level of log reportingand verbosity. The cruises argument can be given infive different variations (under Windows, onlyvariants 1 and 4 are supported—install the pro-grams under Cygwin or Windows Services forUNIX to bypass this limitation):
1.
It can be one or more 8-character NGDC cruisenames.
2.
It can be one or more 2-character NGDC sourceinstitution codes. This selects all the cruisessubmitted by the specified institutions.
3.
It can be one or more 4-character codesconsisting of a 2-character source institutioncode followed by a 2-character vessel code; thiswould return all the cruises from the specifiedsource institution collected by the chosen vessel.
4.
It could be a filename with the prefix ‘‘ ¼ ’’prepended. Then, the file is expected to contain alist of cruise arguments (one per line) of the typesspecified in 1–3 above.
5.
It can be left blank, which tells the program toselect all the cruises in the data base.
Thus, to convert the MGD77 format file01020019.mgd77 (Lamont-Doherty’s famed Eltanin
19 cruise) to a netCDF file we simply run
mgd77convert 01020019�Fa�Tþc�V
The log report settings will ensure that allproblems found during conversion will be reportedto standard output, which can be redirected to a logfile. Because the MGD77+ format is netCDF-based and adheres to the COARDS convention, the01020019.nc file can be examined by the general-purpose java application ncBrowse (available fromhttp://www.epic.noaa.gov/java/ncBrowse). This ex-tends the usefulness of the new format considerably(Fig. 2).
The current MGD77 format with its implieddecimal point places limits on the precision of thestored data. For instance, all magnetics and gravityvalues are stored to nearest 0.1 nT and 0.1mGal,respectively, whereas msd is stored to nearest meter.With these limitations, the five fields faa, eot, mag,diur and msd may be stored as 2-byte integers in theMGD77+ format. However, by using the –4 optionin mgd77convert we can instead use 4-byte integersfor these quantities with a stored precision of10�4 nT, 10�5mGal, and 10�5m, respectively (sui-table when the original data precision improvesbeyond the present MGD77 limits). All other itemsare stored using precisions that far exceed thecurrent precision of marine data.
3.2. Obtaining information about cruises
The tool mgd77info examines the selectedMGD77 files and reports on metadata stored inthe file headers or prepares statistical summariessuch as the extent of a cruise in space and time andthe number of geophysical observations of eachkind. It also can determine which data columns arepresent for each cruise, thus allowing users tofind cruises with certain combinations of data.
where the options control what kind of reporting isdesired. For instance, to see a formatted version ofall the header metadata information for 15040215(Scripps cruise Roundabout, leg 10 on the R/V
Washington), use
mgd77info 15040215�Mf
whereas to list the departure ports of all cruises inthe database, try
mgd77info�MfPort_of_Departure
The names of all the 66 metadata components aredescribed in the mgd77info documentation.
3.3. Augmenting cruises with additional data
The program mgd77manage allows you to addadditional data columns to your MGD77+ files.These can contain any data, including text strings,but most likely are numerical values sampled alongthe track from a supplied grid or an existing column
that has been filtered or manipulated for aparticular purpose (other data sources are unlikelyto be co-registered with your track navigation). TheMGD77+ format supports up to 32 such extracolumns; see the mgd77manage documentationpage for details on how to add columns. You maylater decide to remove some of these columns orupdate the data associated with a certain column.Data extraction tools such as mgd77list (describedbelow) can be used to extract a mix of standardMGD77 columns (navigation, time, and the usualgeophysical observations) as well as your customcolumns.
Given the availability of gridded data sets such asglobal bathymetry and gravity as well as regionalgrids of various kinds, it is often useful to samplesuch grids along a ship track. This will allow fordirect comparisons between different data sets aswell as simplifying data quality control. Further-more, scientists sometimes prefer to process certaindata along-track, e.g., apply a high-pass filter to themagnetic anomaly, and it would be convenient tostore the modified observations within the originaldata file. Finally, in quality control it is helpful tocompare the archived values of reference fields and
corrections to newly computed values based on themetadata. The tool mgd77manage was designed tohandle all these cases. It can append new columnsbased on a variety of sources, including tables ofobservations given as functions of time or distancealong track, sampling of gridded data files alongtrack (including the Mercator grids utilized foraltimetry-derived products (Sandwell and Smith,1997; Smith and Sandwell, 1997)), and the creationof reference data such as the magnetic and gravi-metric reference fields or Carter-corrected depths.The new columns are added to the netCDF-versionof the cruise and become accessible as regular datacolumns. The synopsis of mgd77manage is
where the –A option selects what kind of data are tobe added, the –I option provides the information forthe new column, while the other options controltechnical aspects such as how distances are calcu-lated and interpolations are performed. Only theoptions for sampling grids, calculating correctionterms, or deleting columns allow for more than oneNGDC cruise to be specified at a time. For instance,to sample the ETOPO5 global bathymetry gridalong the track of cruise 29020004 (a 1972 Soviet-era cruise on the vessel Vityaz), we run
mgd77manage� Ietopo5=‘‘ETOPO5
bathymetry’’=m=s=1=0=‘‘1m precision’’ 29020004
�Agetopo5:grd�V
where we use the GMT gridfile etopo5.grd and storethe extracted data as 2-byte integer meters. The newdata column is labeled etopo5. If we want to add acolumn that represents the IGRF magnetic refer-ence field for the time-period of this cruise, stored ina single-precision floating point format, we can run
mgd77manage 29020004�Acm
�Iigrf =‘‘IGRF magnetic field’’=nT=f=1=0=‘‘IGRF 10’’�V
where the IGRF total field intensities are auto-matically calculated from the navigation and theIGRF10 model coefficients (Maus et al., 2005).
3.4. Extracting data from cruises
The versatile utility mgd77list can extract almostany combination of data from the MGD77database. It can compute differences betweenrelated columns on the fly as well as only outputthose data records that satisfy certain logicalexpressions. It can also apply corrections and skipdata whose bitflags suggest they are suspect. Dataoutput may be in plain ASCII or binary single/double precision floating points. The synopsis formgd77list is
mgd77list cruises�Fdataflags½; tests�
½�Acjdjfjg½code�� ½�Cfjgje� ½�Dastartdate�
½�Dbstopdate� ½�E� ½�H�½�Iajcjt�
½�L½corrtable:d�� ½�M½flag�� ½�N½sjp�½ejkjmjn��
½�Rwest=east=south=north� ½�Sastartdist½unit��
½�Sbstopdist½unit�� ½�T½mje�� ½�V�
½�Wweight� ½�Z½þj�� ½�bo½sjdjSjD��
with several options for selecting which data andderived quantities (Table 3) to extract and placevarious limits on the extraction process. Thedocumentation page lists numerous examples ofpossible usage; here we will simply illustrate twoexamples: to obtain an ASCII table with latitude,longitude, and free-air anomaly from all cruises inthe list cruises.lis, we run
mgd77list ¼ cruises:lis�Flat; lon; faa
4grav_data:dat
whereas we can obtain a table of locations withoriginal (i.e., not interpolated) depth measurementsexceeding 9000m anywhere in the world by runningthe command
mgd77list�F’lon; lat; depth;DEPTH
49000;BTC! ¼ 1’4really_deep:xyz
Here, the argument to –F has been placed insingle quotes to avoid having the exclamation point(! ¼ means not equal to) used to test the BathymetryType Code be interpreted by the UNIX shellinstead.
3.5. Locating a particular file
Since MGD77 data may reside in multipledirectories, the mgd77path utility will return thefull path to a particular cruise, or perhaps list all the
ARTICLE IN PRESS
Table 3
Derived quantities available in mgd77list
Name Content
carter Carter table correction to add to uncorrected depth
dist Along-track distance in m, km, miles, or nautical miles [see –C and –N]
azim Azimuth of vessel at each point of observation [see –C]
igrf International Geomagnetic Reference Field 10, valid for 1905–2010
ngrav Normal gravity field, based on one of four selectable reference fields
Heiskanen 1924, International Gravity Field 1930, 1967, or 1980
vel Vessel speed in m/s, km/h, miles/h, or knots [see –C and –N]
directories that have such files. The synopsis formgd77path is simply
mgd77path cruises�DjP½�Iajcjt� ½�V�
where you select a listing of either directories or fullpaths. For instance, to find the full path to thecruise 03020036 (a 1985 National Ocean Servicecruise on the Surveyor), use
mgd77path 03020036
3.6. Plotting cruise tracks on maps
As MGD77 data are often used in conjunctionwith GMT, we supply the tool mgd77track whichdraws the tracks of selected cruises on a map. Beinga supplement of GMT, all standard GMT mapprojections and plot options are accessible. Thesynopsis of mgd77track is
mgd77track cruises�Rwest=east=south=north
�Jparams ½�A½c�½size�� ½�Btickinfo�
½�Cfjgje� ½�Dastartdate� ½�Dbstopdate�
½�Iajcjt� ½�K� ½�Ltrackticks� ½�N� ½�O� ½�P�
½�Sastartdist½unit�� ½�Sbstopdist½unit��
½�TTjtjdms;mc;mfs;mf ;mfc� ½�U½dx=dy=�½label��
½�V� ½�Wpen� ½�Xx_shift� ½�Yy_shift� ½�cncopies�
The options select the cruises and limit the databased on time and distances along track, set up theplot domain and specify how the track should appearon the map, how often and what kind of annotationsshould accompany each track, and allow specifica-tion of attributes such as pen thicknesses and color.As an example, we make a global track map of allcruises contributed by the University of Hawaii (code08) using a Robinson projection:
pscoast�Rg�JN180=9i�B30
�Gbrown�K4map:ps
mgd77track 08�R�J�O�Vbmap:ps
3.7. Performing along-track quality control
While most users are likely just to use the data assupplied by NGDC, expert users and in particularstaff at source institutions responsible for preparingnew MGD77 files to be submitted to NGDC maywant to ensure their data pass the numerous sanitychecks implemented in the quality control applica-tion mgd77sniffer. This tool will optionally comparethe along-track data to corresponding locations inavailable grids of gravity and bathymetry. Thesynopsis for mgd77sniffer is
The options specify what outputs the user wants,the kinds of checks to be applied, and for globalgrids to be specified, implying an along-tracksampling of the grid and comparison to ship-basedobservations. While any grid may be used, theglobal gravity and bathymetry solutions of Sandwelland Smith are particularly useful, hence mgd77snif-fer supports both geographic and Mercator grids. Itwill perform a thorough along-track sanity check ofthe original MGD77 ASCII files and produce acorresponding error log (i.e., the errata) which weidentify with the file extension ‘‘.e77’’. All problemsfound are encoded in the error log, and recom-mended fixed correction terms are given, if needed.The analyst may verify that the suggested correc-tions are indeed valid (we only want to correct trulyobvious scaling errors and outliers), then modify thecorrection terms if necessary and activate them bychanging the relevant code key (see the mgd77snif-fer documentation for more details).
Before we discuss how to detect and correct errorswe will first list the different classes of errors knownto be associated with MGD77 data.
4.1. Error categories
1.
Header record errors occur when some of theinformation fields in the header do not complywith the MGD77 specification or requiredinformation is missing. These errors typicallydo not affect the data and are instead errors inthe metadata.
2.
Fixed systematic errors occur when a particulardata column, despite the MGD77 specification,has been encoded incorrectly by the sourceinstitution. This usually means the data will beoff by a constant factor such as 10 or 0.1 or, ashas been documented in some cases of bathy-metry, even 1.8288, which converts fathoms tometers (Smith, 1993).
3.
Unknown systematic errors occur when theinstrument that recorded the data or the proces-sing that followed introduced signals that appearto be systematic functions of time along track,latitude, heading, or some other combination ofterms that have a physical or logical explanation.These terms may sometimes be resolved by dataanalysis techniques such as along-track (Chand-ler and Wessel, 2004) and across-track (Wessel,1989) investigations, and will result in correctionterms that, when applied to the data, will removethese unwanted signals in an optimal way.Because these correction terms may change asnew data are considered in their determination,such corrections are considered to be ephemeral.
4.
Individual data points or sequences of data mayviolate rules such as being outside of possibleranges or in other ways violate sanity. Further-more, sequences of points that may be withinvalid ranges may give rise to data gradients thatare unreasonable. The status of every observa-tion can therefore be determined and this givesrise to bitflags GOOD or BAD.
Some data points or sequences may pass thesetests, yet later be shown to be in conflict with newdata. These data are marked as questionable and amechanism is needed to exclude them should theuser require it.
The policy we will adhere to states that errorsources 1, 2, and 4 will be corrected by supplying theinformation as meta-data in the relevant *.nc files,whereas the corrections for error sources 3 and 5(because they will constantly be improved) will bemaintained in a separate list of corrections. Thesecorrection tables can then be queried when theprograms access the data.
4.2. Errata description
The errata log file produced by mgd77sniffercontains a variable-length header section followedby any number of errata records, each summarizingerrors encountered in one MGD77 record. Informa-tion pertaining to an entire cruise, such as NGDCand survey institution identification codes, cruiseexamination time, two-way travel time correctorinformation, data precision warnings, and systema-tic scales and DC shifts from global grid compar-isons are reported as E77 header information.Individual errata records follow a strict format:they contain a time (or distance for cruises withouttime) tag followed by the record number, aformatted error code string, and finally a human-readable description of the errors detected in therecord. Four error classes are encoded into the errorcode string with different alphabetic charactersrepresenting unique error types. Errata records usethe following syntax:
time=distance� �
record_number� �
error code string� �
verbose description� �
The following is a few representative errata records:
The error code string is comprised of four errorclasses including navigation errors (NAV), invaliddata values (VAL) (either out of range or flat-lined),excessive data gradients (GRAD), and excessiveoffsets from grids (GRID). These error classes havealphabetic codes describing unique error types. TheNAV error class has six unique cases whilealphabetic characters describe the VAL, GRAD,
and GRID class errors for each of 15 non-navigation numeric fields in the MGD77 file.Table 4 lists the various errors and the defaultsettings for acceptable ranges of values and theirgradients; the user can override these limits ifneed be.
In addition to the other duties discussed earlier,the mgd77manage tool can ingest these errata filesand (1) correct bad header records given therecommendations in the log, (2) insert fixed correc-tion terms to be used when reading certain columns,and (3) insert the bit-flags found for individualobservations. Re-run this step if you later find other
Table 4
Errata codes used by mgd77sniffer
Code Field NAV or VALa
All
0 all OK
Navigation
A t Time out of range
B t Decreasing time
C t,lat,lon Excessive speed
D lat,lon Above sea level
E lat Lat undefined
F lon Lon undefined
Values
A drt 3 or 5
B tz �12 to + 13h
C year 1939 to current year
D month 1–12
E day 1 to last day in mont
F hour 0–24
G min 0–60.0
H lat See Navigation
I lon See Navigation
J ptc 1, 3, or 9
K twt 0–15 s
L depth 0–11,000m
M bcc 1–55, 59–63,88, 98–99
N btc 1, 3, or 9
O mtf1 19,000–72,000 nT
P mtf2 19,000–72,000 nT
Q mag 71000 nT
R msens 1, 2, or 9
S diur 7100 nT
T msd �1000 to +11,000m
U gobs 975,900–986,000mGa
V eot 7150mGal
W faa 7500mGal
X nqc 5, 6, or 9
aError if a value z is outside stated range(s).bError if Dz/Dd (distance in km) or Dz/Dt (time in s) exceed stated gcError if z exceeds grid observation (here used only for depth and fa
problems, as all flags will then need to be recreatedor augmented.
The extraction program mgd77list allows forthese corrections to be applied on-the-fly as dataare requested. First, data with BAD bitflags aresuppressed. Second, data with fixed systematiccorrection terms are corrected accordingly. Third,data with ephemeral correction terms will havethose corrections applied (if a correction table issupplied). All of these steps require the presence ofrelevant metadata and the user can overrule eachstep. In addition, users may add their own bitflagsas separate data columns and use mgd77list’s logical
tests to further dictate which data are suppressedfrom output.
4.3. Detecting and managing systematic errors
As an example of using mgd77sniffer, we scanand correct data from the 1977 USGS cruiseL1077BS on the Lee as archived at NGDC:
mgd77sniffer 06050025
�gfaa; =data=GRIDS=grav:15:2:B:img; 0:1; 1
�Rg�Wx
This initial step allows the mgd77sniffer to examinehow well ship-board free-air anomalies compare withthe specified satellite altimetry-derived global gravitygrid using Reweighted Least Squares regressionanalysis (Rousseeuw and Leroy, 1987). Regressionslope and intercept of 1 and 0 are expected,respectively. Other detected data patterns are re-ported, such as lower than expected data precision.Output from the above command is presented below:
06050025 (faa) Slope 0.088 is statistically differ-ent from 106050025 (faa) RLS scale: 0.088 offset: �0.48 r:0.92 significant: 1 decimation: 006050025 (faa) Max ship-grid difference [�108.3]at record 8802
The analyst examines the output noticingapprox-imate regression scale of �0.1 in free-air anomaly(faa). After thorough inspection of cruise data, theanalyst determines the recommended free-air scaleof 10 to be reasonable, and also noticed that allgravity values after 10:46 on October 16 are invalid,probably as a result of gravimeter malfunction orbad processing. We thus specify an option that flagfaa and gobs as unusable for this range of records;these adjustment are encoded as follows:
mgd77sniffer 06050025
�gfaa; =data=GRIDS=grav:15:2:B:img; 0:1; 1
�Rg�De�Afaa; 10; 0
�Ifaa; 7019; 8812�Igobs; 7019; 8812
This command generates the following ‘‘E77’’errata log file (only the first few lines shown):
# Cruise 06050025 ID L1077BS MGD77 FILEVERSION: 19870604 N_RECS: 8812# Examined: Wed Mar 15 08:25:51 2006
# Examiner: pwessel# Arguments: -gfaa,grav.15.2.img,0.1,1 -Rg -Afaa,10,0 -Ifaa,7019,8812# -Igobs,7019,8812 -De# Errata: HeaderY-faa-E-01: Regression scale 0.841 different from1. Recommended: [10]N-faa-E-02: Regression offset -4.77 differentfrom 0. Recommended: [4.8]Y-faa-E-05: Record range with invalid data:[7019-8812]Y-gobs-E-05: Record range with invalid data:[7019-8812]I-faa-W-04: Integer precision in faa# Errata: Data1977-10-10T19:00:00 11 0-0-W-0 GRAD: faaexcessive1977-10-10T19:04:00 15 0-0-W-0 GRAD: faaexcessive
The corrections implied by the errata file06050025.e77 can then activated by updating the
MGD77+ data file thus:
mgd77manage 06050025�Ae�V
The errata files can either be in the currentdirectory or in the subdirectory E77 in the directorypointed to by the environmental variableMGD77_HOME. Fig. 3 shows both the originaland modified data compared to satellite-derivedfree-air anomalies. Note that while the factor of 10corrects the mistake of storing faa using incorrectunits, it is likely further corrections, such as anoffset, will be identified when the cruise is comparedto others in a crossover analysis, and someindividual points are likely to be outliers (of errortype 5). Finally note that, in this example, thecorrected data necessarily are integer values as anysub-mGal precision in the original data was lostwhen there were erroneously scaled to mGal priorto storage in the MGD77 file.
An important component of our NGDC qualitycontrol project is to produce E77 errata files for allNGDC cruises so as to correct all errors of type 1, 2,and 4. It is hoped that these will eventually beprovided via NGDC but will for the time being behosted from the mgd77 web site at SOEST. Erratafiles will be posted as they become available.
4.4. Managing ephemeral corrections
For ephemeral errors (type 3) we prefer to keepthe correction terms separate from the data files
ARTICLE IN PRESS
-100
-50
0
50
100
150
Rel
ief (
m)
-40
-20
0
20
40
Fre
e-ai
r A
nom
aly
(mG
al)
11 12 13 14 15 16 17October 1977
Original
-100
-50
0
50
100
150
Rel
ief (
m)
-40
-20
0
20
40
Fre
e-ai
r A
nom
aly
(mG
al)
Corrected
no gobs
Fig. 3. Time-series showing free-air anomaly from NGDC cruise 06050025 (NOAA cruise L1077BS from 1977) and altimetry-derived
free-air anomaly from Sandwell and Smith’s grid. (lower) Original data are shown which clearly are off relative to satellite gravity, and
furthermore all faa and gobs values after 10:46 on October 10 are erroneous as gravimeter was either turned off or malfunctioned. (upper)
Scaling original data by 10 gives a much better fit, strongly suggesting data were erroneously stored in mGal instead of 10�1 mGal units.
Data values considered invalid by mgd77sniffer have been excluded. Grayshaded shapes represent bathymetry from ETOPO2.
themselves. If a table of such corrections is availablewe may use the –L option in mgd77list to makecorrections using the provided information. Forcomplete flexibility, mgd77 corrections are assumedto be of the general form
Dz ¼Xn
i¼1
ciffunci½siðzi � oiÞ�gpi , (1)
where each correction can be expressed as a sum ofn terms; Dz will then be subtracted from theobservation z before output. For each term i, funci
(if given) is a function such as sin, cos, and exp thatoperates on the argument given in the brackets. Usescale si and origin oi to translate the argument zi. Ifzi is time then we recognize an origin of T to mean‘‘the start of the cruise’’. The result in the braces isthen raised to the power pi and finally multiplied bythe given amplitude ci. For most corrections (suchas correcting for a overall scale factor or a linear
trend with time), the expressions simplify consider-ably since no functions will be given and allpi ¼ si ¼ 1. The correction table may provide theseparameters for all cruises and observations thathave had correction terms evaluated. The format ofthe correction table mimics Eq. (1) in that each linefollows a simple format
ngdc� id field term1 term2 . . . termn
where field is the abbreviation of the observationthat needs a correction. Each term is of the form
factor � ½cos j sin j exp�ð½scale�ðfield½�origin�ÞÞ½^power�
where the square brackets and what is inside themare optional. All other symbols, including theparentheses, are required, except for any constantterms in which only factor needs to be specified. Todemonstrate the format of the correction table, letus consider a hypothetical cruise 12345678whose depth, faa, and mag need the following
This means the free-air anomalies need to beadjusted for a linear drift term, the depth wassomehow Carter corrected twice and we need toremove the correction once, while there was some-thing funny in the magnetics that turned out to be afunction of the ship’s heading. The correspondingthree lines in the ASCII correction table are:
12345678 faa 12:31:165e� 5�ðtime� TÞ
12345678 depth� carter
12345678 mag 10:3�0:18� cosðazim� 22Þ^2
For more details on the format of the table, seethe mgd77list documentation. Typically, such cor-rection terms are established following exhaustivecrossover analysis of the data (e.g., Wessel andWatts, 1988; Smith, 1993). When the x2sys supple-ment is fully operational it will facilitate thedetermination of such terms as well as the format-ting of the required correction tables.
5. Initiating a local MGD77 database
The starting point for installing your ownMGD77+ database is the MGD77 ASCII datafiles distributed from NGDC on DVD-ROMS andvia FTP. As files are compressed on the distributionmedia, we recommend using NGDC’s suppliedGeodas program to install the files locally. Further-more, we choose the ‘‘Carter corrected depth’’option, which will fill in the depth column usingthe two-way travel times and the Carter tables iftravel times are present. This step yields �5000individual cruise files, requiring about 7.3Gb ofdisk space. You should place these in one or moresub-directories of your choice, then list these sub-directories (one per line) in the file mgd77_paths.txt,and place that file in the directory pointed to by theenvironment variable $MGD77_HOME; if not setthis variable defaults to $GMTHOME/share/mgd77. For our setup, we chose to place the filesin separate directories according to source:
#$Id: mgd77_paths.txt,v 1.4 2005/09/05 06:10:51pwessel Exp$## Directories with MGD77 files
The first three directories are the source directoriesfor MGD77 files (with the most recent data directorylisted first), whereas the MGD77+ directory is wherewe intend to place the converted files.
Once the Geodas extraction has been successful,we change directory to enter the MGD77+directory where we will run mgd77convert since itwill write the converted files to the current directory.We convert all the MGD77 files to the netCDFMGD77+ format using the command
mgd77convert�Fa�1Tc�V�Lwe
þ4conversion: log
With almost 5000 files to convert, this is a goodtime to take your lunch break. When completed, theMGD77+ directory will have ‘‘.nc’’ versions of allfiles, with a total size of only 2.3Gb. Although noinformation has been lost, and no compressionapplied, the change in format from ASCII to binaryand the avoidance of repeating fields that remainconstant for all records (or are absent) yields almosta 70% savings in disk space. Unless you plan to usethe MGD77 files in different applications you maydelete them or compress them to reclaim disk space.
Acknowledgments
The IGRF calculations are based on a Fortranprogram written by Susan Macmillan, BritishGeological Survey, translated to C using f2c byJoaquim Luis, University of Algarve, Portugal, andadapted to GMT style by us. This work wassupported by US NSF grant OCE-0241590. SOESTcontribution number 6814.
References
Becker, J.J., Sandwell, D.T., 2006. Global estimates of seafloor
slope from single-beam ship soundings. EOS Transactions of
the American Geophysical Union 87 (36) Ocean Science
Meeting Supplement, Abstract OS23L-04.
Calmant, S., Berge-Nguyen, M., Cazenave, A., 2002. Global
seafloor topography from a least-squares inversion of
