1

Integration of the Eurostat and ESS Metadata Systems

A. Götzfried Head of Unit B6

Eurostat

2 2

OutlineOutline

The Eurostat Metadata-Handler and two of its main The Eurostat Metadata-Handler and two of its main componentscomponents

Euro SDMX RegistryEuro SDMX Registry National Reference Metadata Editor (National RME)National Reference Metadata Editor (National RME)

The standard structures for metadata reportingThe standard structures for metadata reporting

Euro SDMX Metadata Structure (ESMS)Euro SDMX Metadata Structure (ESMS) ESS Standard for Quality Reports structure (ESQRS)ESS Standard for Quality Reports structure (ESQRS)

The Generic The Generic Statistical Business Process Model (GSBPM)Statistical Business Process Model (GSBPM)

Impacts on the sImpacts on the statistical business processestatistical business processes

3 3

The Eurostat Metadata-Handler

is the backbone of a metadata-driven harmonised statistical business process at Eurostat and in the European Statistical System;

provides one central interface for accessing different types of metadata by Eurostat staff or by ESS members;

provides services to other IT applications within the ESS;

is the primary source for different types of harmonised metadata to be used within Eurostat and the ESS.

4 4

The Eurostat Metadata-Handler and its main components

National Metadata Editor

National Metadata Editor

EMISEMIS

RAMONRAMON CODEDCODED

Eurostat Metadata Handler

Common user interface

Eu

ro S

DM

X

Reg

istr

y

5

The Euro SDMX Registry as part of the Eurostat-MH

The Euro SDMX Registry currently contains : The Euro SDMX Registry currently contains :

• the Eurostat and ECB Data Structure Definitions in use for GESMES-based data transmission;

• the code lists currently used in the different GESMES-based Data Structure Definitions;

• the list of data and metadata flows (from countries to Eurostat); • the list of statistical concepts used in the abovementioned data

structure definitions; • some additional information such as provision agreements, etc.

The Euro SDMX Registry will contain in the future: The Euro SDMX Registry will contain in the future:

• the agreed SDMX data and metadata structure definitions used in different statistical domains for data and metadata exchange;

• the harmonized code lists used in SDMX data structure definitions; • national data structure definitions and code lists used in the ESS

countries (if uploaded by countries).

6

The use of the Euro SDMX Registry for National Statistical Authorities

The National Statistical Authorities will have access to the Euro SDMX Registry :

• for storing the data and metadata structure definitions they are using at national level;

• for storing the code lists they are using at national level;

• for retrieving the SDMX data and metadata structure definitions used within the ESS or beyond;

• for retrieving the harmonised structural metadata (code lists) used within the ESS or beyond;

• for retrieving information related to data and metadata flows and other standards (the concepts used, the provision agreements, etc… )

7

The National Reference Metadata EditorThe National Reference Metadata Editor : :

• is the component dealing with national reference metadata;• will accommodate the current ESS standards for reference

metadata such as the ESMS or the upcoming ESQRS; • is addressed to the national producers of reference metadata,

who will be able to compile their domain-specific metadata on-line and transmit them to Eurostat;

• is the IT tool stimulating the harmonization of reference metadata within the ESS;

• will be made available to the ESS later in 2010.

The National Reference Metadata Editoras part of the Eurostat-MH

8

Business process related to a national reference metadata file

9 9

The standard structures for Metadata reporting

Progress made since 2008 :

2008: introduction of the Euro SDMX Metadata Structure (ESMS) at Eurostat and the ESS for the production and dissemination of reference metadata (Commission Recommendation 498/2009)

01/2009: release of the new version of the ESS quality reporting documents:

• ESS Standard for Quality Reports (ESQR)• ESS Handbook for Quality Reports (EHQR)

Detailed requirements following the European Statistics Code of Practice ESS Quality and Performance Indicators (QPI’s) defined

03/09: EP/Council Regulation 223/2009 Article 12 defining the quality criteria to be reported

10 10

The Euro SDMX Metadata Structure (ESMS)

The 21 concepts and sub-concepts of the ESMS :

11 11

The Euro SDMX Metadata Structure (ESMS)

The ESMS is more oriented to the USERS of statistics:

helps to understand the statistical data released does not include too detailed information on data quality uses 21 SDMX cross domain concepts now attached at upper level on Eurostat’s website

12

The new ESQRS Based on the ESQR (ESS Standard for Quality Report), a new

report structure - the ESS Standard for Quality Reports Structure: ESQRS - was created for harmonising the reporting on statistical data quality within the ESS.

The ESQRS is using the main statistical data quality criteria as listed in EP/Council Regulation 223/2009 and as being part of the ESMS and details them further :

• Relevance• Accuracy• Timeliness and Punctuality• Accessibility and Clarity• Comparability• Coherence

A subset of the Quality Performance Indicators (QPI’s) is covered in the new ESQRS

13

From ESQR to ESQRS

XI. Conclusions

X. Confidentiality,

IXI. Performance, Cost and Respondent

Burden

VIII. Assessment of User needs and

Perceptions

VII. Trade-offs between Output Quality Components

XI. Conclusions

X. Confidentiality,

IXI. Performance, Cost and Respondent

Burden

VIII. Assessment of User needs and

Perceptions

VII. Trade-offs between Output Quality Components

ESQR ESQRS

Transparency, Security

III ACCURACY

IV. TIMELINESSand PUNCTUALITY

V. ACCESSIBILITYand CLARITY

VI. COMPARABILITYand COHERENCE

II. RELEVANCE

I. Introduction to the Statistical Process

and Its Outputs

IntroductionII IntroductionII

Relevance (user needs and perceptions)

III Relevance (user needs and perceptions)

III

AccuracyIV AccuracyIV

Timeliness and punctualityV Timeliness and punctualityV

Accessibility and clarityVI Accessibility and clarityVI

ComparabilityVII ComparabilityVII

CoherenceVIII CoherenceVIII

ContactI ContactI

CommentIX CommentIX

14

ESMS and ESQRS

ESQRS is more oriented to the PRODUCERS of statistics to monitor the quality of the statistics produced in detail concentrating on the main quality concepts (being also part of the ESS Statistics Regulation No 223/2009)

The information in the ESQRS is more detailed than the information on quality criteria in the ESMS but there is consistency and information common to both ESMS and ESQRS

ESQRS goes deeper than ESMS in terms of quality criteria

15

ESMS and ESQRS

Accuracy and reliability Non- sampling error

Description:Accuracy:closeness of computations or estimates to the exact or true values that the statistics were intended to measure.Reliability: closeness of the initial estimated value to the subsequent estimated value.

ESMS

Description:Error in survey estimates which cannot be attributed to sampling fluctuations.

Non- response errorNon- response error Unit response rateUnit response rate Formulae unit resp. rateFormulae unit resp. rate

Description:The difference between the statistics computed from the collected data and those that would be computed if there were no missing values.

Description:The ratio of the number of units for which data for at least some variables have been collected to the total number of units designated for data collection.

Description:

Ex. calculation formluae for un-weighted unit response rate.

Accuracy Non- sampling error

Description:Accuracy:closeness of computations or estimates to the exact or true values that the statistics were intended to measure.

Description:Error in survey estimates which cannot be attributed to sampling fluctuations.

ESQRS

16

The Generic Statistical Business Process Model (GSBPM)

17

The GSBPM in Eurostat : the Eurostat CVD

18

Impacts on the statistical business processes

The Eurostat Metadata Handler is SDMX compliant and impacts the statistical business process at Eurostat and within the ESS:

– The Euro SDMX Registry will make available all SDMX harmonised artefacts (Code lists, DSD’s, MSD’s, etc.) to Eurostat and the ESS

– The National RME will offer a way to harmonise the collection, structure, content and format of national reference metadata and national quality reports

Development of ESS common building blocks – Eurostat works on a standardised process metadata structure

19 19

All over

Good progress of the Eurostat Metadata Handler since 2008 but Good progress of the Eurostat Metadata Handler since 2008 but work has to be pursued in harmonisation and integration of the work has to be pursued in harmonisation and integration of the statistical business processesstatistical business processes

Implementation of the ESS metadata standards is essential to:Implementation of the ESS metadata standards is essential to:– enable a more automatic production, exchange and dissemination of enable a more automatic production, exchange and dissemination of

metadatametadata– minimise the reporting burden for statistics producersminimise the reporting burden for statistics producers– improve availability and clarity for statistics usersimprove availability and clarity for statistics users– promote for international exchange of informationpromote for international exchange of information– reduce the costs and burden in Eurostatreduce the costs and burden in Eurostat– reinforce the reporting according to EP/Council Regulation 223/2009reinforce the reporting according to EP/Council Regulation 223/2009

The harmonisation work also considerably contributes to the The harmonisation work also considerably contributes to the improvement and integration of the statistical business processes improvement and integration of the statistical business processes as defined in the Commission Communication 404/2009. as defined in the Commission Communication 404/2009.