MoreInformation/Feedback:•DataandproductsareacquiredaspartofNASA'sEarthScienceDataSystemsandarchivedanddistributedbytheCrustalDynamicsDataInformationSystem(CDDIS):

C.Noll,TheCrustalDynamicsDataInformationSystem:Aresourcetosupportscientificanalysisusingspacegeodesy,AdvancesinSpaceResearch,Volume45,Issue12,15June2010,Pages1421-1440,ISSN0273-1177,DOI:10.1016/j.asr.2010.01.018.

•ThestaffwelcomesfeedbackontheCDDISandinparticulartheideasexpressedinthisposter;contactPatrickMichael(Patrick.Michael@nasa.gov).

1:NASAGoddardSpaceFlightCenter,Greenbelt,MD,USA2:Stinger-Ghaffarian Technologies,Lanham,MD,USA3:Science,SystemsandApplications,Lanham,MD,USA

CrustalDynamicsDataInformationSystemBackground:•IsNASA’sactivearchiveofspacegeodesydata,products,gatheringinformationfrom:

Ø GlobalNavigationSatelliteSystem(GNSS)Ø SatelliteLaserRanging(SLR)Ø VeryLongBaselineInterferometry(VLBI)Ø DopplerOrbitography andRadio-positioningIntegratedbySatellite(DORIS)

•Collectingdatafromover1000sitesworldwide.•Providingdataarchiving&distributionsupportforoverthirtyyears.•ThelargestCDDISusercommunitycomesfromtheserviceswithintheInternationalAssociationofGeodesy(IAG).•CDDIShasbeenanESDISDAACsince2007

ArchiveStatistics:•Archivesize:~17.5TB•Ingestrate:~9GB(100Kfiles)/day•Distributionrate:~500GB(~4.7Mfiles)/day•Data(L1,L1B),products(L2)andinformationaboutdataandproducts•Latenciesrangingfrom:multi-day,daily,hourly,sub-hourly,andsub-minute•Expecteddistributionin2016is1.5Bfiles,180TBandover140,000usersserved

OperationsOverview:•CDDIS’sprimaryoperationscanbecategorizedas:

Ø Providesystemstocollectdata& derivedproductsfromworldwidesuppliersØ Performqualitycontrolandmakedata&productsavailablefordistributionØ Collect&storemetadata&providedatadiscoverybothinternally&externallyØ Developvalued-addedreportsfromarchivecontentsforsciencecommunity

•CDDIS’sarchiveanddistributionhavebeengrowingatover30%annually•CDDIStransitionedtoallnewinfrastructureandimplementedanentirelynewGNSSprocessingsystemon1December2016.

GNSSProcessing:Overview:•GNSSaccountsfor95%ofallincomingfiles• ~30millionfiles/year• Deliveryratesupto100files/second•Latencyfromreceiptofdatafilestoavailabilityinthearchiveiscommonlylessthan30seconds•FilescannotjustbecopiedtothearchivebutMUSTbequalitytestedbeforemakingavailable

IngestOperations:•Allnewfileuploadprocess.SeeposterG41B-1013 andhttps://cddis.nasa.gov/About/CDDIS_File_Upload_Documentation.html•Allfilesundergoabatteryofsecuritychecksbeforeanyprocessingtakesplace.•24totalqualitycontrolchecks•5warningchecks– somecanbefixedinreal-timeExamplesinclude:

Ø IncorrectcompressionschemeØ IncorrectfiletypesubmittedØ Incorrectnamingscheme

•19“fatal”checks– errorsthatstopfileprocessingExampleincludes:

Ø UnknownfiletypeØ FilenamedoesnotmatchfilecontentsØ UnknownRINEXversionØ UnknownmonumentmarkerØ Virusinfectedfile

NextGenerationGlobalNavigationSatelliteSystem(GNSS)ProcessingatNASACDDIS

IN43B-1691

ABSTRACT: The Crustal Dynamics Data Information System (CDDIS) has been providing access to space geodesy and related data sets since 1982, and in particular, Global Navigation Satellite Systems (GNSS)data and derived products since 1992. The CDDIS became one of the Earth Observing System Data and Information System (EOSDIS) archive centers in 2007. As such, CDDIS has evolved to offer a broad range ofdata ingest services, from data upload, quality control, documentation, metadata extraction, and ancillary information. With a growing understanding of the needs and goals of its science users CDDIS continuesto improve these services. Due to the importance of GNSS data and derived products in scientific studies over the last decade, CDDIS has seen its ingest volume explode to over 30 million files per year or morethan one file per second from over hundreds of simultaneous data providers. In order to accommodate this increase and to streamline operations and fully automate the workflow, CDDIS has recently updatedthe data submission process and GNSS processing. This poster will cover this new ingest infrastructure, workflow, and current operations.

MetadataOperations:•Allincomingfilesareprocessedforbothfileandcontentmetadata•68totalmetadataparametersaregatheredforeachGNSSfile

Ø Filemetadata– 16parametersperfileØ Contentmetadata- 52parametersperfile

•Content& filemetadataarepublicallysearchableinmultipledatadiscoverytools•MetadataareprovidedtotheEarthObservingSystemDataInformationSystem(EOSDIS)throughtheirCommonMetadataRepository(CMR)fordatadiscoveryacrossall12DistributedActiveArchiveCenters(DAACs).•MetadatawilleventuallybemadeavailabletotheGlobalGeodeticObservingSystem(GGOS)fordiscoverybythegoedetic community

ReportingOperations:•Dataproviderreports

Ø ErrorreportsØ Trendingreportsovertime

• SystemlevelreportingØMetricsgeneration• Ingestrates• Dataprovidererrorrates

Ø Trendingreports•Sciencecommunityreports

Ø GNSSstatusreports

FutureCapabilities:•SystemwilleventuallyincorporateprocessingofSLR,DORIS,andVLBIdatafiles•Additionalcontentmetadatacollections•Automatedingestreportsbydataprovider•Betteranalysisandhandlingofduplicatefiles•Webinterfaceforviewingreal-timeprocessingstatistics•Additionalgeodeticcommunitydirectedreports/summaries•Transitionentireprocesstoanagilesoftwaredevelopmentpath

1PatrickMichael (Patrick.Michael@nasa.gov)1CareyNoll(Carey.Noll@nasa.gov)2JustineWoo (Justine.Y.Woo@nasa.gov)3RebeccaLimbacher (Rebecca.I.Limbacher@nasa.gov)

Formoreinformation:https://cddis.nasa.gov

Files delivered?

Yes

No

End

General File Check

Fatal Errors?Yes

Check File Content and Collect Metadata

No

Fatal Errors?

Push errors to DB

Yes

Warnings?

NoPush

warnings to DB

Process the GNSS Files

Yes

Push Recorded Metadata to DB

Require Additional Processing?

Move Files to Archive

Copy to Additional Processing Directory

Yes

No

Database

File/Content Metadata

Tables

Error Tables

Archive

Error Report:- Duplicate Files

- Fatal errors- Warnings

GNSS Status Files

Flat Files

<driver software>

Check User Directories for FilesCheck Additional Processing

Directory for Files

Files in directory

?

Yes

No

Perform Additional Processing

Files Created?

No Yes

<report software>

CDDISOperationalOverview

SpaceGeodesySiteLocations

GNSSWorkflowDiagram

340795

24464

3642

Pass Warning Fatal

UserReportonDuplicateFiles

ProcessingReports,MetricsandGraphics•Metricsonindividualdataproviders•Systemwidestatistics•Metricsonfilequality

Ø BydataproviderØ Byfiletype

•Metricsonnumber/sizeofincomingfiles

TotalFilesProcessedReportOver6DayPeriodPass=FilesSuccessfullyProcessed

Warnings=FilesCorrectedDuringProcessingFatal=FilesNotAbletoBeProcessedDuetoErrors

TotalFileProcessingOveraSixDayPeriod