SDMX IMPLEMENTATION ASPECTS: GETTING STARTED, PRACTICALITIES AND

MAINTENANCE PROCESS

SDMX Training

B A N K I N D O N E S I A1 6 - 1 8 S E P T E M B E R 2 0 1 5

Y O GYA KA RTA , I N D O N E S I A

T HIS CHECK LIST IS TA K EN F RO M A DRA FT OF A DOCUM EN T UNDER DEV ELOP M ENT BY THE

SDM X STATIST ICA L W ORK ING GROUP

P LEA SE NOT E T HAT THE DOCU M ENT IS ST ILL UNDER DEV ELOP M EN T A ND REV IEW BY THE

SW G

Checklist

How to Model a Statistical Domain for Data Exchange in SDMX (under drafting): These guidelines outline general principles on how to design and create SDMX artefacts in a statistical domain, following a step-by-step approach based on the SDMX information model. The document is also providing recommendations on the number of Data Structure Definitions (DSDs) for a subject-matter domain, following the approach that the decision on the number of DSDs needed for describing a statistical domain should come after - and as a consequence of - the discussion on the data model and the architectural approach to be followed in the data collection exercise. Contact point : SDMX Statistical Working Group at swg@sdmx.org.Guidelines for the Creation and Management of SDMX Code Lists (under revision): These guidelines are intended to support the creation of Codelists to be used all along the statistical business process and in particular when SDMX is implemented in statistical domains. They are strongly recommended for use when SDMX-compliant Data Structure Definitions (DSDs) are built-up and implemented in statistical domains. Contact point : SDMX Statistical Working Group at swg@sdmx.org.SDMX Glossary (under revision): Previously known as the Metadata Common Vocabulary (MCV) the revised SDMX Glossary will be more specifically SDMX-oriented and will cover a) SDMX cross-domain Concepts (i.e. standard concepts which should be used across statistical domains wherever possible to enhance exchange and comparison possibilities), b) the technical terminology from the Implementation Model and other techncial specifications, and c) various important statistical concepts. Contact point : SDMX Statistical Working Group at swg@sdmx.org.Global and/or local registries: See under Section 2. above.Examples of Global DSD matrices: Examples of Global DSD matrices as well as overviews of artefacts used in a Global DSD can be found in existing Global DSDs (e.g. for National Accounts: http://sdmx.org/?page_id=1498, Balance of Payments and Foreign Direct Investment: http://sdmx.org/?page_id=1747.Handbook for Statisticians on how to use SDMX: This document was drafted in 2012 in the framework of an ESSnet contract sponsored by Eurostat. This handbook is aimed at people working in the production services but it is also intended to those metadata experts that are responsible for variable and classification definitions used for data dissemination. It is also addressed to those readers already having some knowledge of the SDMX standard and that want to have only a hands-on approach on data modelling. Any depth investigation on SDMX is left to the official documentation. [Link to be added]

Inspiration taken from the following documents

Lifecycle

1. Initiate1. Initiate

1.1 Acquire a basic understanding of SDMX standards and guidelines, and implementation tools

1.3 Assign roles and responsibilities (preliminary)

1.2 Identify stakeholders (within organisation, across NSIs and at international level)

1.4 Draft a project charter

2. Plan and Organize2. Plan and Organize

2.3 Specify general context of data exchange

2.2 Assign roles and responsibilities

2.1 Organize kick-off meeting

2.4 Develop work breakdown and project schedule

2.5 Miscellaneous planning and organizational issues

3. Design3. Design

3.6 Design derived variables and transformation formulae

3.1 List and describe data flows to be covered

3.2 Elaborate concept scheme

3.3 Build DSD matrix

3.4 Optimize matrix

3.7 Design associated Metadata Structure Definitions (MSDs)

3.5 List artefacts to be developed

4 Develop and Test4 Develop and Test

4.1 Determine fitness for purpose of existing implementations with a view to saving time and resources, and reusing as much as possible existing material (e.g. Concept Schemes, Code Lists)

4.2 Develop SDMX artefacts

4.3 Develop associated Metadata Structure Definitions (MSDs)

4.4 Draft documentation

4.5 Test

4.6 Make corrections

4.7 Review documentation

5 Implement5 Implement

5.1 Upload information to output systems

5.2 Formalize maintenance agreements

5.3 Launch information campaign

5.4 Provide support, training and capacity building

6 Evaluate6 Evaluate

6.1 Gather evaluation inputs

6.2 Conduct evaluation

6.3 Agree an action plan

6.4 Implement action plan

1. Initiate1. Initiate

1.1 Acquire a basic understanding of SDMX standards and guidelines, and implementation tools

1.3 Assign roles and responsibilities (preliminary)

1.2 Identify stakeholders (within organisation, across NSIs and at international level)

1.4 Draft a project charter

One effective way of learning about SDMX and its benefits is to implement a small scale pilot study on a statistical domain using the available global Data Structure Definitions and suites of implementation tools

1. Initiate1. Initiate

1.1 Acquire a basic understanding of SDMX standards and guidelines, and implementation tools

•Consult existing documentation

•One effective way of learning about SDMX and its benefits is to implement a small scale pilot study on a statistical domain using the available global Data Structure Definitions and suites of implementation tools

•Attend webinars and training sessions, capacity building sessions, international conferences

•Install, test and get used to basic tools to produce XML files

1. Initiate1. Initiate

1.3 Assign roles and responsibilities (preliminary)

•Agree on a project manager or project management cell (general project management, organization of meetings, calendar keeper, monitoring and control, reporting, etc.)•Assign roles and responsibilities of all other stakeholders

1.4 Draft a project charter

1.2 Identify stakeholders (within organisation, across NSIs and at international level)

•Senior management (Experience has shown that it is essential that senior management drives SDMX)•Subject-matter specialists•Experts in SDMX standards and technical implementation•Experts in code lists, classifications and methodology•Coordination / quality unit•Data dissemination unit•IT specialists

2. Plan and Organize2. Plan and Organize

2.1 Organize kick-off meeting

•Define project scope•Set expectations of all key project stakeholders•Identify project risks•Discuss project plans

2.2 Assign roles and responsibilities (final)

•Domains: Single, multi•Purpose: Single, multi•Type of data:

data and metadata, metadata onlymicrodata, macrodataone-dimensional data set, multi-dimensional data set, time series

•Level of data exchangewithin organizationbetween national organizationsbetween national and international organizationsbetween international organizations

2.3 Specify general context of data exchange

2. Plan and Organize2. Plan and Organize

2.3 Specify general context of data exchange

•Type of data recipient: human, machine•Location of data recipient•Role in data exchange

collectionproviderDSD maintenanceend user

•Process pattern: bilateral, gateway, data sharing•Data and metadata exchange mode: Push, pull•Data and metadata architecture

data warehouse (repository) architecturehub architecturedata producer architectureregistry

•Frequency of data exchange

2.4 Develop work breakdown and project schedule

•Break down the project into smaller and more manageable components such as deliverables, work packages, activites, and tasks•Project schedule:

identify dependencies between tasksassign resources for each taskidentify task start and end dateswork out the overall project duration

•Distribute plans to stakeholders and discuss them

2. Plan and Organize2. Plan and Organize

3. Design3. Design

3.1 List and describe data flows to be covered

3.2 Elaborate concept schemes

•List all concepts needed for the domain observed•Map/align chosen concepts to SDMX concepts•Consolidate concepts•Group concepts: Cross-domain concepts, shared consepts, specific concepts•Add default representation to concepts (drawing on existing concept schemes)

3.3 Build DSD matrix

•Put data flows in relation to the concept scheme (i.e. which concept is used in which data flow)

•If SDMX implementations have already taken place in neighbouring domains, consider synergy and integration opportunities (e.g. extension of existing DSDs instead of development of new ones)

3. Design3. Design

3.4 Optimize matrix

•Fill blanks in the matrix with default codes

3.5 List artefacts to be developed

3.6 Design derived variables and transformation formulae

•Check which SDMX official artefacts are already readily available (e.g. cross-domain concepts, cross-domain code lists)

3.7 Design associated Metadata Structure Definitions (MSDs)

4. Develop and Test4. Develop and Test

4.1 Determine fitness for purpose of existing implementations with a view to saving time and resources, and reusing as much as possible existing material (e.g. concept schemes, code lists)

4.2 Develop SDMX artefacts

•Derive DSDs from the matrix: 1 DSd per set of dimensionality•Optimize the number of DSDs by single-fixing data flows to a code where required•Develop domain-specific concepts and code lists•Create dataflows•Create constraint artefacts

4.3 Develop associated Metadata Structure Definitions (MSDs)

4.4 Draft documentation

4.5 Test

4.6 Make corrections

4.7 Review documentation

5. Implement5. Implement

5.1 Upload information to output systems

•Upload artefacts to the appropriate registry •Resolve issues•Validate and sign off•Disseminate documentation

5.2 Formalize maintenance agreements

•Draft agreement on the governance of the SDMX artefacts

Provide overview of all artefacts relating to a particular statistical domainProvide details about the maintenance of artefacts over timeMention any other maintenance aspects relevant to te domain concerned

5.3 Launch information campaign8.2 Conduct evaluation (GSBPM 8.2)

•Post announcement on ad hoc website(s)•Make announcements at meetings

5. Implement5. Implement

5.4 Provide support, training and capacity building8.2 Conduct evaluation (GSBPM 8.2)

•Organise training sessions (e.g. webinars)•Provide helpdesk (e.g. functional email address, forum)

6. Evaluate6. Evaluate

6.1 Gather evaluation inputs

•Determine persons / team to conduct evaluation•Gather inputs required for evaluation

6.2 Conduct evaluation

•Conduct detailed analysis and evaluation of all gathered inputs•Produce report detailing finding, and recommendations for improvement

6.3 Agree on action plan 8.2 Conduct evaluation (GSBPM 8.2)

•Present evaluation report to appropriate boards for discussion•Agree on action plan for either implementing or amending the proposed recommendations•Set up metrics to monitor the success and benefits derived from implementing recommendations

6.4 Implement action plan 8.2 Conduct evaluation (GSBPM 8.2)

Proof of Concept

The lead unit within an organisation and / or lead agency within the data exchage community would also take responsibility for organising a proof of concept pilot study that would demonstrate the benefits of SDMX implementation.• Which unit / area within the organisation would be the primary

contact and main driver for the pilot implementation project - e.g. IT unit; training area; subject matter area; data dissemination or coordination units?

The key and most time consuming process for SDMX implementation is the harmonisation of structural metadata that will facilitate the exchange of data.

Reference Metadata

Determine • what metadata currently exists;• whether it is structured and uses a standard metadata template

that covers a number of domains – refer below;• whether the national structural metadata concepts are consistent

with those used in the SDMX Content-Oriented Guidelines, i.e. for the Cross Domain Concepts located in the SDMX Glossary;

• where it is stored. In many countries metadata are frequently stored and disseminated in Word files or via PDFs. 

Options for such a template are the use of: • the template used for the IMF’s Special Data Dissemination

Standard (SDDS);• Eurostat’s ESMS or Single Integrated Metadata Structure; or• a national metadata template.

Questions?