1
Integration of the Eurostat and ESS Metadata Systems
A. Götzfried Head of Unit B6
Eurostat
2 2
OutlineOutline
The Eurostat Metadata-Handler and two of its main The Eurostat Metadata-Handler and two of its main componentscomponents
Euro SDMX RegistryEuro SDMX Registry National Reference Metadata Editor (National RME)National Reference Metadata Editor (National RME)
The standard structures for metadata reportingThe standard structures for metadata reporting
Euro SDMX Metadata Structure (ESMS)Euro SDMX Metadata Structure (ESMS) ESS Standard for Quality Reports structure (ESQRS)ESS Standard for Quality Reports structure (ESQRS)
The Generic The Generic Statistical Business Process Model (GSBPM)Statistical Business Process Model (GSBPM)
Impacts on the sImpacts on the statistical business processestatistical business processes
3 3
The Eurostat Metadata-Handler
is the backbone of a metadata-driven harmonised statistical business process at Eurostat and in the European Statistical System;
provides one central interface for accessing different types of metadata by Eurostat staff or by ESS members;
provides services to other IT applications within the ESS;
is the primary source for different types of harmonised metadata to be used within Eurostat and the ESS.
4 4
The Eurostat Metadata-Handler and its main components
National Metadata Editor
National Metadata Editor
EMISEMIS
RAMONRAMON CODEDCODED
Eurostat Metadata Handler
Common user interface
Eu
ro S
DM
X
Reg
istr
y
5
The Euro SDMX Registry as part of the Eurostat-MH
The Euro SDMX Registry currently contains : The Euro SDMX Registry currently contains :
• the Eurostat and ECB Data Structure Definitions in use for GESMES-based data transmission;
• the code lists currently used in the different GESMES-based Data Structure Definitions;
• the list of data and metadata flows (from countries to Eurostat); • the list of statistical concepts used in the abovementioned data
structure definitions; • some additional information such as provision agreements, etc.
The Euro SDMX Registry will contain in the future: The Euro SDMX Registry will contain in the future:
• the agreed SDMX data and metadata structure definitions used in different statistical domains for data and metadata exchange;
• the harmonized code lists used in SDMX data structure definitions; • national data structure definitions and code lists used in the ESS
countries (if uploaded by countries).
6
The use of the Euro SDMX Registry for National Statistical Authorities
The National Statistical Authorities will have access to the Euro SDMX Registry :
• for storing the data and metadata structure definitions they are using at national level;
• for storing the code lists they are using at national level;
• for retrieving the SDMX data and metadata structure definitions used within the ESS or beyond;
• for retrieving the harmonised structural metadata (code lists) used within the ESS or beyond;
• for retrieving information related to data and metadata flows and other standards (the concepts used, the provision agreements, etc… )
7
The National Reference Metadata EditorThe National Reference Metadata Editor : :
• is the component dealing with national reference metadata;• will accommodate the current ESS standards for reference
metadata such as the ESMS or the upcoming ESQRS; • is addressed to the national producers of reference metadata,
who will be able to compile their domain-specific metadata on-line and transmit them to Eurostat;
• is the IT tool stimulating the harmonization of reference metadata within the ESS;
• will be made available to the ESS later in 2010.
The National Reference Metadata Editoras part of the Eurostat-MH
8
Business process related to a national reference metadata file
9 9
The standard structures for Metadata reporting
Progress made since 2008 :
2008: introduction of the Euro SDMX Metadata Structure (ESMS) at Eurostat and the ESS for the production and dissemination of reference metadata (Commission Recommendation 498/2009)
01/2009: release of the new version of the ESS quality reporting documents:
• ESS Standard for Quality Reports (ESQR)• ESS Handbook for Quality Reports (EHQR)
Detailed requirements following the European Statistics Code of Practice ESS Quality and Performance Indicators (QPI’s) defined
03/09: EP/Council Regulation 223/2009 Article 12 defining the quality criteria to be reported
10 10
The Euro SDMX Metadata Structure (ESMS)
The 21 concepts and sub-concepts of the ESMS :
11 11
The Euro SDMX Metadata Structure (ESMS)
The ESMS is more oriented to the USERS of statistics:
helps to understand the statistical data released does not include too detailed information on data quality uses 21 SDMX cross domain concepts now attached at upper level on Eurostat’s website
12
The new ESQRS Based on the ESQR (ESS Standard for Quality Report), a new
report structure - the ESS Standard for Quality Reports Structure: ESQRS - was created for harmonising the reporting on statistical data quality within the ESS.
The ESQRS is using the main statistical data quality criteria as listed in EP/Council Regulation 223/2009 and as being part of the ESMS and details them further :
• Relevance• Accuracy• Timeliness and Punctuality• Accessibility and Clarity• Comparability• Coherence
A subset of the Quality Performance Indicators (QPI’s) is covered in the new ESQRS
13
From ESQR to ESQRS
XI. Conclusions
X. Confidentiality,
IXI. Performance, Cost and Respondent
Burden
VIII. Assessment of User needs and
Perceptions
VII. Trade-offs between Output Quality Components
XI. Conclusions
X. Confidentiality,
IXI. Performance, Cost and Respondent
Burden
VIII. Assessment of User needs and
Perceptions
VII. Trade-offs between Output Quality Components
ESQR ESQRS
Transparency, Security
III ACCURACY
IV. TIMELINESSand PUNCTUALITY
V. ACCESSIBILITYand CLARITY
VI. COMPARABILITYand COHERENCE
II. RELEVANCE
I. Introduction to the Statistical Process
and Its Outputs
IntroductionII IntroductionII
Relevance (user needs and perceptions)
III Relevance (user needs and perceptions)
III
AccuracyIV AccuracyIV
Timeliness and punctualityV Timeliness and punctualityV
Accessibility and clarityVI Accessibility and clarityVI
ComparabilityVII ComparabilityVII
CoherenceVIII CoherenceVIII
ContactI ContactI
CommentIX CommentIX
14
ESMS and ESQRS
ESQRS is more oriented to the PRODUCERS of statistics to monitor the quality of the statistics produced in detail concentrating on the main quality concepts (being also part of the ESS Statistics Regulation No 223/2009)
The information in the ESQRS is more detailed than the information on quality criteria in the ESMS but there is consistency and information common to both ESMS and ESQRS
ESQRS goes deeper than ESMS in terms of quality criteria
15
ESMS and ESQRS
Accuracy and reliability Non- sampling error
Description:Accuracy:closeness of computations or estimates to the exact or true values that the statistics were intended to measure.Reliability: closeness of the initial estimated value to the subsequent estimated value.
ESMS
Description:Error in survey estimates which cannot be attributed to sampling fluctuations.
Non- response errorNon- response error Unit response rateUnit response rate Formulae unit resp. rateFormulae unit resp. rate
Description:The difference between the statistics computed from the collected data and those that would be computed if there were no missing values.
Description:The ratio of the number of units for which data for at least some variables have been collected to the total number of units designated for data collection.
Description:
Ex. calculation formluae for un-weighted unit response rate.
Accuracy Non- sampling error
Description:Accuracy:closeness of computations or estimates to the exact or true values that the statistics were intended to measure.
Description:Error in survey estimates which cannot be attributed to sampling fluctuations.
ESQRS
16
The Generic Statistical Business Process Model (GSBPM)
17
The GSBPM in Eurostat : the Eurostat CVD
18
Impacts on the statistical business processes
The Eurostat Metadata Handler is SDMX compliant and impacts the statistical business process at Eurostat and within the ESS:
– The Euro SDMX Registry will make available all SDMX harmonised artefacts (Code lists, DSD’s, MSD’s, etc.) to Eurostat and the ESS
– The National RME will offer a way to harmonise the collection, structure, content and format of national reference metadata and national quality reports
Development of ESS common building blocks – Eurostat works on a standardised process metadata structure
19 19
All over
Good progress of the Eurostat Metadata Handler since 2008 but Good progress of the Eurostat Metadata Handler since 2008 but work has to be pursued in harmonisation and integration of the work has to be pursued in harmonisation and integration of the statistical business processesstatistical business processes
Implementation of the ESS metadata standards is essential to:Implementation of the ESS metadata standards is essential to:– enable a more automatic production, exchange and dissemination of enable a more automatic production, exchange and dissemination of
metadatametadata– minimise the reporting burden for statistics producersminimise the reporting burden for statistics producers– improve availability and clarity for statistics usersimprove availability and clarity for statistics users– promote for international exchange of informationpromote for international exchange of information– reduce the costs and burden in Eurostatreduce the costs and burden in Eurostat– reinforce the reporting according to EP/Council Regulation 223/2009reinforce the reporting according to EP/Council Regulation 223/2009
The harmonisation work also considerably contributes to the The harmonisation work also considerably contributes to the improvement and integration of the statistical business processes improvement and integration of the statistical business processes as defined in the Commission Communication 404/2009. as defined in the Commission Communication 404/2009.