How to implement an SDMX infrastructure for dissemination and … 2013 Session 7.4 - How to...

1 11-13 Sep 2013 SDMX Global Conference 2013 OECD, Paris

How to implement an SDMX

infrastructure for dissemination

and reporting

Session 7: Capacity building – technical stream

Bengt-Åke Lindblad – Eurostat

Francesco Rizzo – Istat


Agenda

Overview of Eurostat's SDMX Reference Infrastructure

Tutorial on how to set up an SDMX architecture through

Eurostat SDMX Reference Infrastructure and related "plug-

ins"

Installation of tools, mapping of statistical concepts,

transcoding between code lists…. etc)

Use of tools acting as client for access, browsing and

presentation of data and metadata


Overview of the

SDMX

Reference Infrastructure


What is the SDMX-RI

Universal framework for modern data provision and exchange

Set of pick-and-choose reusable building blocks allowing a

statistical office to expose data to the external world based on

access rights

Designed to provide data and structural metadata based on

mappings to each organization's dissemination data warehouse

Uses SDMX standards incl. one for Web Services


SDMX-RI building blocks

Web Service - a service that accepts both SDMX Query

messages for data and structural metadata.

Web Client - a web application that allows users to create

basic SDMX queries to be used for retrieving data from a

Mapping Store database and dissemination databases using a

SDMX-RI Web Service instance.

Mapping Assistant - provides mapping information between

SDMX artefact and the local data storage scheme.

Test Client - meant to test the SDMX-RI building blocks and

the Web Service endpoint.


Why use SDMX-RI

Developed to simplify the exchange of data

Provides standard software and components, allowing

individual statistical organisations to interact and exchange

their data using the same software and methodology

Modular approach, use part or the entire infrastructure, extend

it by adding new modules or modify it in any other way to suit

their own purposes

Developed in both Java and .NET


Where is the SDMX-RI used

As of Sept 2013 deployed in 23 EU countries

Tested across EU Member States (2011 population census)

Running in Mexico

Expression of interest:

The Caribbean, Latin America, OECD and Russia

Autumn 2013:

Expected to run in EU28 Member States

Expected to run in EFTA countries

In DSWS (Eurostat dissemination web service)


SDMX-RI development strategy 2013 -2014

“Systems must continually be adapted to the changing environment; otherwise

their utility will progressively decline”.

Architectural changes (two step approach)

Intermediate solution to solve performance & out of memory errors

"Ultimate" solution to provide a common API and implement

SDMX 2.1

Implement new user requests and correct defects

Widen the scope and usage of SDMX-RI among data

providers

ESS.VIP-programme such as the ICT project

Reuse for other statistical data collections

DSWS (Eurostat dissemination web service)


WS extension to SDMX 2.1 new interfaces

Controller

(3)Data Retriever

(streaming)

Web

Ser

vic

e

Pro

vid

er

(6)Common

SDMX API

(1)Structure

Retriever

NSI_Service_2.0 NSI_Service_2.1

SOAP

Request 2.0

SOAP Request

2.1

NsiRestService

REST Request

2.1

(7)SDMX API

Implementation

(7)SDMX 2.0

Implementation

(7)SDMX 2.1

Implementation


SDMX-RI development timeline

Jan.2013 Feb Mars Apr. May Jun. July Aug. Sept. Oct. Nov. Dec. 2013 |Jan.2014

JAVA SDMX-RI ultimate

API Integrated v 0.9.25

SDMX v2.0 compliant


API v 0.9.25

+ compression

SDMX v2.1 compliant

.NET SDMX-RI ultimate

API v 0.9.25

+compression

SDMX v2.1 compliant



SDMX v2.0 compliant

Mapping Store SDMX v2.1 compliant

.NET/Java SDMX-RI ultimate

+ enhancements

SDMX v2.1 compliant



SDMX v2.0 compliant

.NET SDMX-RI intermediate

+enhanced performance and

compression

SDMX v2.0 compliant

JAVA SDMX-RI intermediate

+compression

SDMX v2.0 compliant


+ enhancements

SDMX v2.1 compliant


How to create an SDMX infrastructure

for dissemination and reporting

Step by step live demonstration


Steps to be executed during the live demo

1. Present the scenario as part of the demonstration (some datasets

already pre-mapped for the demonstration purposes)

2. Use “client” tools to interact with the Web Service (SDMX-RI) to

browse, present and extract data and structural metadata. This action is

performed using the following tools:

SDMX Structural Metadata Browser (developed by Istat)

Web Client application (part of the SDMX-RI)

SDMX MS Excel Plug-in (developed by Istat)

3. Use the Mapping Assistant (SDMX-RI) and Test Client (SDMX-RI)

to prepare a new dataset (Production in industry) to be exposed in

SDMX through the Web Service (SDMX-RI)

4. Use “client” tools, like in point 2, to browse, present and extract the

new dataset


The scenario

Mapping

Assistant

Test Client

Dissemination

database

(.STAT DB

schema)

Mapping

Store DB

Web

Service

SDMX

Structural Metadata

Browser

Web Client

Excel

Plug-in


Mapping Assistant workflow

STEP 2 Load SDMX

Structures

(CategoryScheme,

DataFlow, DSD…)

STEP 4 Map local

Concepts

to SDMX ones

(Mapping Set)

STEP 5

Map local

Codes

to SDMX ones

(Transcoding)

STEP 3 Map local

database

schema

(Dataset)

Query

Editor

Custom

Query

DB Tables

STEP 1

MSDB Connection

DDB Connection


Live demonstration (The slides that follow will only be used in case of technical problem)


Step 1 – MSDB connection


Step 1 – DDB connection


Step 2 Load SDMX Structures


Step 3: Map local database schema (Dataset) -

Query Editor


Step 3: Map local database schema (Dataset) –

Custom Query


Step 4: Mapping local Concepts to SDMX ones

Local concepts SDMX concepts

- FREQ

ATECO_2007 ACTIVITY2007

ITTER107 AREA_EE

CORREZ ADJUSTMENT

TIPO_DATO7 IND_TYPE

- BASE_PER

- MEASURE

VALUE OBS_VALUE

TIME TIME_PERIOD


Step 5: Mapping of local codes to SDMX ones

(transcoding)

Code Description

R Row

D Seasonally, not correct

C Calendar

Y Seasonally and correct

Code Description

N Row

S Seasonally, not correct

W Calendar

Y Seasonally and correct

SDMX CL_ADJUSTMENT

Local CL_ADJUSTMENT

Local SDMX Description

R N Row

D S Seasonally,

not correct

C W Calendar

Y Y Seasonally

and correct


Questions & Answers

Date post:	05-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

How to implement an SDMX infrastructure for dissemination and … 2013 Session 7.4 - How to...

Documents