PDS/IPDA presentation to the IVOA · PDS/IPDA presentation to the IVOA Steven Joy, Baptiste...

transcript

PDS/IPDApresentationtotheIVOAStevenJoy,BaptisteCecconi

Outline• PlanetaryDataSystem(PDS)activities

• PDSOverview• PDSArchiveProcess• PDS4ImplementationanddifferencesfromPDS3• PDS4ImplementationofDOIs• PDS4registryandsearch

• InternationalPlanetaryDataAlliance(IPDA)activities• IPDAOverview• IPDAStructure• IPDAGoalsandChallenges• IPDAProgressandFutureDirections

• SolarSystemInterestGroup(SSIG)• Comments

• Summary

PlanetaryDataSystem(PDS)Overview• PDSis,andhasalwaysbeenforemostadataarchiveandsecondarilyadatasystem

• Asanarchive,PDSrequiresthatdataanddocumentationarestoredinstandard,non-proprietaryformats

• ThisoftenprecludestheuseofcommonuserformatssuchasMSWordfordocumentsorIDLsavesets

• PDSisanopenarchive,availabletoanyone,anywhere• PDSisorganizedontheprincipalthatplanetarydataarediverse,andthatthebestcuratorsofsuchadiversearchivearedomainexperts(scientistandtechnologists)inthevariousfieldsofplanetaryscience.Assuch,PDSisafederationofScienceDisciplineandSupportNodes:

• AtmosphericSciences(ATM)– NewMexicoStateUniv.– NancyChanover• Geosciences(GEO)–WashingtonUniv,St.Louis– RayArvidson• ImagingandCartography(IMG)– USGSFlagstaff– LisaGaddis• PlanetaryPlasmaInteractions(PPI)– UCLA – RayWalker• RingMoonSystem(RMS)– SETI –MarkShowalter• SmallBodies(SBN)– Univ ofMd.,CollegePark – James(Gerbs)Bauer

• NavigationandAncillaryInformation(NAIF) – JPL – ChuckActon• Engineering(ENG)– JPL – DanCrichton• Management(MNG)– GSFC – TomMorgan

• MostoftheScienceDisciplineNodeshavemultiplesub-nodesthatprovideexpertiseacrossmanyaspectsofthedisciplinethatarenotavailableattheNodelocation

PlanetaryDataSystem(PDS)– SomeHistory• PDS1wentoperationalin1989

• Highlevelcatalogforfindingdatasetsbymission,instrument,spacecraftandtarget• ObjectDescriptionLanguage(ODL)isinventedforproductlabelingandcataloginformation• Dataarchivedanddistributedon9-tracktape

• PDS2(1990)• Cataloginformationstreamlined,longertextdescriptions,fewerkeyword/values• CD-ROM(CD-R)becomestheprimaryarchiveanddistributionmedia

• PDS3(1992)• PDSsetsupandmaintainsawebpresence• Movementtoonlinedistribution(~2002)ofelectronicvolumes(PDS-D)• OnlinemassstorageanddatabricksreplaceCD/DVDasarchivedistributionmedia

• PDS4(2014)• Prototypebuild1in2010,operationalin2014withLADEEasfirstmissiontousenewstandard• ModeldrivenarchitecturewithXMLlabels• Distributedregistriesandsearchservicesreplacethecatalogandsupportdirectfilelevelaccess• Newdatastandardsandreducedsetofarchivedataformatsandstructures• Internationalcollaboration

PDSArchiveProcess

• ThePDSarchiveprocessshouldbeaniterativeprocessbetweentheNodeandthedataprovider(s)duringwhichthecontentandstructureofthearchivearenegotiated,delivered,reviewed,andrevised

1. ArchiveDesign– ProvidersproposeacollectionofdataanddocumentationforarchiveandthenworkwiththePDSNodetorefinethedesign.Documentationiswritten,sampleproductswithlabelsaredeliveredforinternalreview.

2. PeerReview– OncetheNodeanddataprovideragreethatthedataarereadytoarchive,aformalpeerreviewissetupwiththereviewpanelconsistingofatleasttwoindependentsciencereviewers,PDSstandardsreviewers,andthedataproviders.Thereviewresultsinoneofthreeoutcomes:• Archived:Dataanddocumentationareacceptedforarchive– noissuesfound,noliens.• CertifiedwithLiens:Dataarecertifiedasscientificallyusefulastheyarebuttherearesomeissuesthat

needtobefixedbeforetheycanbearchived.Datacanbereleasedtothepublicintheinterim.• Notcertified:Dataordocumentationhavedeficienciesthatpreventthemfrombeingscientificallyuseful

untiltheissuesarecorrected.Archivewillneedanotherreviewbeforebeingreleasedtothepublic.3. LienResolution– Thedataprovidercorrectsissuesreportedduringreviewandresubmitsthedatafor

archive.4. Archive – Allissuesreportedduringthereviewhavebeenaddressed.Onlineversionhasusagewarnings

removedandthearchiveissenttotheNSSDCAfordeeparchive.5

InitiateContact

DataProvider

ArchiveDesign

DataProvider

PDSNodeInternalReview

PDSNode

PeerReviewScientistsLiens

ArchiveLienResolution

Verified

Comments

PDS4:AnInternationalPlanetaryArchivingEffort

• PDS4 is an international planetary archive data model and standard that is being used to build compatible archives

• An explicit information architecture• All planetary data tied to a common model to improve validation and discovery

• Use of XML, a well-supported international standard, for data product labeling, validation, and searching.

• A hierarchy of data dictionaries built to the ISO 11179 standard, designed to increase flexibility, enable complex searches, and make it easier to share data internationally.

• An explicit software/technical architecture• Distributed services both within PDS and at international partners

• Consistent protocols for access to the data and services

• Deployment of an open source registry infrastructure to track and manage every product in PDS

• A distributed search infrastructure

PDS4DifferencesfromPDS3

SlidetakenfromD.CrichtonpresentationonOperationalReadiness– 2013-09-137

PDS4DataModel

ThePDS4datamodelgrowsfromtheconceptofadataset,itscontents,andhowiscanbedescribedanddocumentedinasearchabledatasystem

Adatasetconsistofcollectionofdigitalobjects(data,documents,calibration,etc.)thatwhendescribedbymeta-data(labels)becomedigitalproductsAdditionalmetadataarerequiredthedescribetheconceptualobjects(missions,targets,etc.)inordertosupportsearches

CollectionProducts

BundleProducts

Collection ProductsCollection Products

BasicProducts

PDS4ArchiveOrganization

• Thereare3primarytypesofproductsinPDS4:

• Basic Products are the smallest unit of a PDS4 archive. They consist of an individual label and the associated file or files.

• Related basic products of the same type may be grouped together into a Collection.

• Related collections may be grouped together into a Bundle.

10100001000100101001101101000100001

SlidetakenfromJ.Mafi PDS4trainingpresentation,April20189

PDS4LogicalandVersionIdentifiers• ALogicalIdentifier(LID)isauniqueIDthatmaybeusedtoidentifyandreferenceanyPDS4product.

• LIDsmustbeuniqueacrosstheentiredatasystem• LIDstaketheformofaUniformResourceName(URN)• LID:urn:nasa:pds:bundle:collection:product

• urn,agency,andorganizationarestatic,butmayvarybyarchivingorganization(e.g.“urn:esa:psa”,“urn:jaxa:darts”,etc.)

• bundle isabundleidentifier(e.g.“maven-swea-calibrated”)• collection isacollectionidentifier(e.g.“data-svy-pad”)• productisanidentifierfortheindividualproduct

• TheproductVersionIdentifier(VID)maybeappendedtotheLIDtoformaLIDVID• Adoublecolon(::)isthedelimitertoseparatetheVIDfromtheLID• LIDVID:urn:nasa:pds:bundle:collection:product::vid

• WhenaproductisrequestedusingonlyaLID,themostrecentversionoftheproductisreturned.SpecificversionsofproductscanberequestedbyspecifyingthefullLIDVID.

FundamentalDataStructures

PDS4 archive products must be describable using one of the following fundamental structures:

Array – homogenous binary structures of 1 to 16 dimensions in which all of the elements have the same data type.

Table – ASCII or binary data with a repeating record structure made up of fixed-width fields.

Parsable Byte Stream – ASCII data with a repeating record structure made up of variable width fields separated by a field delimiter (e.g. CSV).

Encoded Byte Stream – Files formatted according some established standard (e.g. PDF).

PDS4Implementation

PDS4XMLSchema

PDS4XML

SchematronPDS4

Metadata

The structure and content of PDS4 metadata is defined by a formal Information Model.Information Model

PDS4 is implemented in XML and expressed in terms of XML Schema and Schematron files.

SchemaSchematron

• Schema define the metadata structure• Schematron provide rule-based constraints on

elements and content

AnatomyofaPDS4Label

• XMLidentificationtag;Schematron identification(optional)XML Declaration XML Declaration

Product (Root) Tag

Identification Area

Observation/Context Area

File Area

Reference List

Product Tag• Root tag; Namespace declarations; Schema identificationIdentification Area• Product identifying information

Observation/Context Area• Product provenance/background

• Links to relevant products and publications

• File format and/or structural information

Reference List

File Area

PDS4UseofDOIs• ThemostrecentversionofthePDS4datamodeldoesnotyetsupporttheuseofDOI’s

• Thechangerequestforincludingthisattributewasstillunderreviewatthefreezedateofthelastbuild.

• DOI’swillbeincludedinthenextbuildthatwillbecomepublicinOctober2018

• PDSiscurrentlygeneratingDOIsforallPDS4collections,andforselecteddocumentsanddataproducts

• DataarchiveSIS(softwareinterfacespecification)andcalibrationproceduredocumentswillbeassignedDOIs– listsofprojectpersonnelwillnotbe

• OnlyspecialdataproductsthatareexpectedtobesitedintheliteraturefrequentlyarelikelytobeassignedDOIs– nodataproductstodatetomyknowledge

• PDSisregisteringDOIswiththeU.S.Dept.ofEnergyOfficeofScientificandTechnicalInformation(OSTI)

• DOIscanonlybeassignedbyLIDVIDsincetheyrefertoaspecificversionofaproductordocumentthatmustalwaysbevalid

• ADOImaybereservedforacollectionordocumentsothatitcanbeincludedinalabelpriortorelease(publication) 14

MetadataHarvestingandRegistration

InternationalPlanetaryDataAlliance

InternationalPlanetaryDataAlliancehttp://planetarydata.org

MissionofIPDA*

• Facilitateglobalaccessto,andexchangeof,highqualityscientificdataproductsmanagedacrossinternationalboundaries

• Supportconstructionofcompatiblearchives• Supportsharingoftoolsandsoftwareservices• DefinedatastandardswithintheIPDA,includingthedata

modelsandderiveddictionaries,basedontheNASAPlanetaryDataSystem(PDS)thatisthede-factostandardforallplanetarydataatthetimeoftheIPDAfounding

*ExtractedfromIPDACharter,July2007

SteeringCommitteeMembers

Structure

28Members12Countries/InternationalInstitutionsChair:TomStein,WashingtonUniversityDeputy Chair:ChristopheArviest,ESA

Technicalexpertsgroup

~20Members

Generalactivities

Annualmeeting,usuallyinJulyorAugust,20-30participantsRegularteleconferencesevery3months10-20participantsParticipationinrelatedmeetings:COSPAR,EGU,AGU,EPSC,etc...IPDAWebsitehttps://planetarydata.org

IPDAGoalsandProgress1. Supportconstructionofcompatibleplanetarysciencedataarchives

• In2012,theIPDAendorsedPDS4asthearchivingstandardforplanetarydata• Implementation,oraplantodoso,ofthePDS4standardacrossagenciesformissionarchiveincludingESA,IKI,ISRO,JAXA,KARI,NASA,UAE.

2. Supportthesharingoftoolsandsoftwareservices• ThePDS4validationtoolususedbyseveralIPDAmemberswhichimprovesinteroperabilitybetweenagencies.

• TheToolRegistryisanotherexampleofavirtualclearinghouseofplanetarydatarelatedtoolscreatedbyavarietyofproducersacrosstheglobe.

3. Facilitateglobalaccessto,andexchangeof,highqualityscientificdataproductsmanagedacrossinternationalboundaries• Atpresent,thereareRESTbasedaccessservicesareinplacebetweenESA,ISRO,andNASA.• High-levelsearchbetweenESAandNASAhasbeendemonstratedandisinplaceforbothPDS3andPDS4archives.

• ProjectsareunderwaytodevelopcitationlinkagewithpublicationsandtocreatebridgesbetweentheIPDAandtheInternationalVirtualObservatoryAlliance(IVOA).

IPDAFutureChallenges1. Supportconstructionofcompatiblearchives

• Guidelinesonhowtobuildhighqualityarchives(contentandreview)• Improvedsupportanddocumentationincludingtutorialsforimplementation

2. Supportsharingoftoolsandsoftwareservices• ServiceregistryforGIStechnologiesandtoolsassuggestedatthePlanetaryGISmeeting• Coordinatedtooldevelopmentacrossinternationalboundaries

3. Facilitateglobalaccessto,andexchangeof,highqualityscientificdataproductsmanagedacrossinternationalboundaries• APIaccesstoarchives• Continuetodevelop,evolve,andpromotetheOpenPlanetaryDataAccessProtocol(PDAP)servicesatJAXAincludingwebmappingservices

• IntegrateISROdataintointernationalsearchandaccess• Productlevelsearchandaccessacrossagencies• Integrateaccesswithcomputation• Expandinteroperabilityacrossspacesciences

SolarSystemInterestGroup(SSIG)oftheIVOACharter/Goals

• IreviewedtheSSIGcharterandtheonlyrealsuggestionthatIhaveisthatthestatementbeamended,probablyinthe“Goals”area,toaddastatementaboutself-promotion

• Somethinglike:

Promoteawarenessinthescientificcommunityoftheongoingeffortsoftheastrophysics,heliophysics,andplanetarysciencesdisciplinedatasystems/archivestosupportsysteminteroperabilityanddataexchange.

• Withoutpublicawarenessandsupport,itwillbedifficulttohaveoureffortsfunded• Valuetothecommunityneedstobedemonstratedbyouractionsandcommunicatedtoourpeersthroughthecoordinatedeffortsofourmembers

SolarSystemInterestGroup(SSIG)oftheIVOAReferenceFrames

• I’dliketostronglyendorseaddingsupportforcommonplanetaryreferencesframes• Keyissuesarenamingconventionsandthecommunicationofdefinitions

• Currentlywithintheplanetarycommunitytherearemultiplenamesforthesameframe• Therearealsoframesthatappeartobethesamebuthavesubtledifferencesthatarenotwellcommunicatedinthenamingconventionsand/ordescriptions

• NAIF/SPICEiscurrentlyintheprocessofformallydefiningandnamingawiderangeofdynamicplanetary/satellitereferenceframesthathavebeenpreviouslyusedinarchivingordefinedintheliterature.

• Thiseffortimposesauniformityinnamingconventions• Formalmathematicalframedefinitions• Documentationoftheoriginalframereferencesandnames(i.e.aliaslistsformanyframes)

• IftheNAIF/SPICEframenames/IDsbecomerecognizedastheproperwaytouniquelyspecifyplanetaryframes,andtheyareadoptedbythearchivesandrecordedinthemetadata,thenwemayhaveapathforwardfortheexchangeofframeinformationbetweendatasystems

• I’mnotsuggestingthattheIVOAcommunityadoptoruseSPICE,onlythatweconsideringleveragingtheSPICEefforttoformalizeframenamingconventions

Summary• PDSisalongstandingNASAorganizationresponsiblefordataarchive• PDS4isanewinformationmodeldrivenapproachtomodernizingthearchivestandardsusedwithintheplanetarysciencecommunity

• PDS4isbeingdevelopedwithinternationalparticipation• ThroughtheIPDA,memberorganizationscanmakestandardschangerequeststothePDS4datamodelandhaveavotingmemberonthechangecontrolreviewboard

• Designedtosupportinteroperabilitybetweendistributedplanetarydataarchivesystems

• IPDAisaninternationalorganizationofplanetarydataarchiveswithmembersfromatleasttwelvecountries

• IPDApromotestheadoptionofcommonstandards,cross-systemsearchandretrieval,andthesharingofsoftware,tools,andmethodsofdataexchange

• TheSSIGisafairlyneworganizationwithintheIVOAthegoalofworkingwithotherIVOAgroupstoreview,assess,andproposelow-impactadjustmentstotheIVOAstandardstosupportsolarsystemsciences

• Standardizinglistsofcoordinatesystemsandreferenceframesisoneofseveralareaswherethisgroupcanhaveapositiveimpact 23

PDS/IPDA presentation to the IVOA · PDS/IPDA presentation to the IVOA Steven Joy, Baptiste...

Documents