1

Metadata Working Group Meeting 2012

Summary and conclusions

August Götzfried and Håkan Linden

Unit B5

2

Item 3: ESS vision and implementation actions

• ESS business process integration and the implementation of the ESS vision is a medium and long term process taking up several years.

• Many ESS actions are undertaken for creating and implementing ESS technical and statistical standards as well as harmonised ESS IT infrastructure and IT applications. Also the harmonisation and integration of ESS legislation progresses.

• We need a strong buying-in to this strategy from Member states. Further implementing this strategy means that the ESS will get more integrated over time. After an investment phase this should lead to overall resource savings for producing and disseminating European statistics.

3

• Eurostat has just started to work on E.A. to be the basis for streamlining the implementation of the Joint ESS strategy.

• The context (role, value added, architectural layers, implementation activities etc.) was explained.

• The E.A. is to be seen as a framework for developing the ESS activities in a dynamic and more transparent way.

• It is defined in a functional (activity driven) way where IT, methodology, metadata are to support the main business processes.

• NSI's and Eurostat should have more common infrastructure to share, process and re-disseminate information.

• The information has to be organised (need for common models, standards etc. that are reusable and compatible).

• Continue with bigger and fewer ESS initiatives/projects with strong business cases (wave of new ESS VIPs).

• E.A. governance needs to be deployed at the ESS level.

Item 4: The ESS Enterprise Architecture (E.A.)

4

• This EP/Council Regulation is an "umbrella regulation" and standard setting, not requiring immediate use of the standards as such (the latter are to be included in implementing measures).

• Standards for micro data exchange are included.• A series of implementing measures (Commission Regulations)

linked to the processes, standards and metadata are planned and will be successively introduced covering SDMX, ESMS, ESQRS etc.

• Version 3 of the EP/Council Regulation will be submitted to the ESSC meeting of 09/2012.

• Plan to start the formal legal procedure in the fourth quarter 2012, i.e. the draft legal act will be sent to the EP and the Council.

• Supporting Community grants (1,3 Mio €) are planned for the third quarter 2012.

• An extension of the time planning beyond August 2012 for submitting the grant proposals will be difficult to realize.

Item 5: ESS legislation on processes, standards and metadata

5

• The Generic Statistical Information Model (GSIM) is a reference framework for modelling information on the process.

• It complements the generic statistical business process model, GSBPM.

• GSIM is a layered model (allowing model at high level and object level).

• It ensures consistency with existing standards.• GSIM v.0.4 available now (for public consultation until 15 June 2012).• Future work will focus on further model specification (mainly at

detailed level), use cases, governance, communication etc.• Four task teams are set up (common workshop in September 2012).• GSIM v.1.0 should be achieved by the end of 2012.• Further information on the UNECE METIS wiki; PT will contribute to

the future work.• Eurostat to provide feed-back in particular on the relationship with the

existing technical standards (e.g. SDMX).

Item 6: GSIM and Sponsorship on Standardisation

6

• The Sponsorship on Standardisation will advise the ESSC on priorities, supporting actions and on the organisation of standardisation and integration in the ESS.

• It runs from 2011-2013 and 6 countries + Eurostat are in the project.• It is co-chaired by CBS and Eurostat.• Three pillars of activity: stocktaking of current standards; framework

for standardisation and pilot studies.• Issues to tackle: to define what a standard is; what standards do we

need; and which ones to prioritise for the ESS business processes?

• More information:http://www.Essnet-portal.eu (preparatory ESSnet on Standardisation)

https://webgate.ec.europa.eu/fpfis/wikis/display/essmethodology/Sponsorship+on+Standardisation (the current Sponsorship on Standardisation)

Item 6: GSIM and Sponsorship on Standardisation

7

• Eurostat presented the National Reference Metadata Editor (NRME) and the user management module: SAM.

• The NRME is an SDMX compliant web application for the production of national reference metadata; it accommodates the current ESS standards for reference metadata (ESMS and ESQRS).

• The SAM module is to be used by National Administrators of the NRME for giving access to national users.

• The NRME is in full use in the ESS since May 2011.• Countries asked to be notified when new metadata flows are opened

up, when metadata files are sent and when new users are added.• More information related to the national users was asked (when

added and by whom, for which flows they have access etc.)• Several countries asked for open up the NRME also for the

production of metadata for national purposes.• Several training and information events (mainly web-based) planned

for National Administrators in 2012-2013.

Lunchtime presentation of the NRME

8

• Objective of the project: to enhance the Eurostat dissemination chain through rationalisation, new functionalities and new technology.

• The VIP runs from 2011- 2013.• Action 4 targets the enhancement of the Eurostat

dissemination towards better integration with MS's dissemination (mainly with regard to common visualisation tools and SDMX compliant web services).

• Cooperation with Member states mainly through the Dissemination Working Group and specific task force meetings.

• Only European statistics will be included in the widgets.• The IT application for the widgets is not decided yet (we are

also looking at national visualisation tools).

Item 7: The VIP Enhanced Dissemination Chain

9

Item 8: Harmonised structural metadata (code lists)

• Currently, 134 ESS SCL created and released by Eurostat on RAMON and in the future also on the Euro SDMX Registry (version 2 of autumn 2012) (with full synchronisation).

• Standard code lists originally used for dissemination, are more and more used also for data production and exchange.

• Eurostat would like that more ESS SCL are also taken over by the SDMX Content-oriented Guidelines.

• Guidelines drafted for the creation and management of SDMX cross domain code lists (strong requirement by SDMX community) with the basic principles for creating and maintaining code lists.

• Countries are invited to send comments on these guidelines as soon as possible.

• Several questions clarified with regard to the principles for the harmonisation work, languages used for the definitions, progress in specific statistical domains etc.

• Work has also started in the SDMX Statistical WG to develop more code lists since the guidelines are now available.

10

Item 9: CODED/RAMON

• RAMON is part of the ESS Metadata Handler used by Eurostat to disseminate various categories of metadata.

• 136 classifications currently uploaded.• The Statistical Concepts and Variables (CODED) database

is a repository of concepts and variables (together with definitions) used within the ESS.

• 10366 terms in EN, 7277 in FR and 6073 in DE available in the CODED production database. Also 962 legal acts, 412 manuals and the ESS Standard Code lists are included.

• About 1000 statistical variables to be released successively (direct links to the data sets to be established).

• Export of Classifications in SDMX format investigated.• Further integration of RAMON with other ESS MH

applications is ongoing.• Member States were encouraged to look at the reusability

of CODED/RAMON, before developing their own metadata/classification repositories.

11

• 330 Eurostat ESMS files under production and dissemination in 2011/2012.

• The Eurostat ESMS files will be the entry point for dissemination of the corresponding national ESMS or ESQRS files.

• 300 Quality Profiles and 140 Environmental Factsheets are in the process of being converted to ESMS (= ESMS Indicator Profiles, ESMSIP).

• Eurostat provides support for converting the existing metadata collections into the ESS standards and into the NRME.

• 37 new national workflows opened in 2012 and six more planned. • 1st Newsletter sent to NRME National Administrator on 25 May 2012.• It is planned to integrate better the ESS Metadata Handler applications

such as EMIS and NRME. • All these actions will rationalise the production and improve the quality

of reference metadata in the ESS.• Member States were invited to send proposals for enhancements of

the NRME.

Item 10: Reference metadata

12

• The metadata in the 2011 round of population and housing censuses are even more important than before given new ways of exchanging the information within the ESS.

• The metadata requirements are in particular regulated in (EC) 519/2010.

• ESMS will be used for the general national metadata and disseminated on the Eurostat website. The NRME will be used for metadata exchange.

• The more detailed metadata on quality will most probably be converted into an (extended) ESQRS and made available through the census hub.

Item 11: Reference metadata in the census hub

13

• The Task Force is set up in order to implement the Sponsorship on Quality recommendations.

• It defined the Single Integrated Metadata Structure (SIMS) and will update the ESS Standard and Handbook for Quality Reports from 2009.

• The SIMS is a conceptual listing of concepts with the aim to ensure consistency between different report structures (such as the ESMS and the ESQRS). The work should be ready by 09/2012.

• A methodological manual will be produced to support the use of the SIMS concepts.

• The outcome of this work will be incorporated into the ESS MH, in particular the NRME.

Item 12: Task Force on the Quality Report Structure

14

• More and more statistical domains implement SDMX when standardising their statistical production processes, for both data (creation of data structure definitions etc.) and metadata (metadata structure definitions, standard code lists etc.).

• The Eurostat SDMX implementation process covers 4 phases: preparation, compliance, implementation and production.

• The implementation process also takes into account data sets shared with other (international) organisations and data exchange rationalisation.

• A set of IT tools are offered to support the SDMX implementation in Member States.

• When a business process is to be redesigned, we take the opportunity to implement SDMX.

Item 13: SDMX implementation

15

• The Euro SDMX registry is part of the ESS Metadata Handler; it acts as a repository for SDMX artefacts (DSD's, code lists, MSD's etc.) used for data and metadata exchange by Eurostat and Member States.

• It has a web-based user interface as well as web-services. It interacts with many other applications such as eDAMIS and the Global Registry.

• It can be downloaded and used in national environments for national purposes.

• It is currently being re-designed: extension of functionalities, improved graphical user interface, support of the SDMX 2.1 technical standards etc.

• Version 2 will be available in the fourth quarter 2012; the content will then be cleaned and responsibilities for the artefacts stored can be broader distributed.

• Eurostat will offer web-training to countries, also on the Euro SDMX Registry when the new version is ready.

Item 14: SDMX registries

16

• The Global Registry complements the Euro SDMX Registry for global SDMX artefacts.

• The SDMX Sponsors will put it in place for storing SDMX global artefacts.

• The SDMX Global Registry will be linked to other SDMX Registries (also the Euro SDMX Registry).

• The governance of the SDMX artefacts is discussed by the SDMX Secretariat (e.g. with regard to maintaining agency, versioning etc.); guidelines will be available in autumn 2012.

• Overall NSIs should access the Euro SDMX Registry which will also provide access to the SDMX Global Registry.

• We will communicate more on the use and access of the Euro SDMX Registry and the Global Registry in the ESS Metadata Handler Newsletter.

Item 14: SDMX registries

17

• The informal SDMX/DDI dialogue was initiated in 2010.• The SDMX Secretariat, DDI Alliance and many other

experts and stakeholders are involved.• The dialogue aims at avoiding duplication of effort by the

two technical standards; to reassure to the user communities of both DDI and SDMX that the end-to-end statistical process can be managed; and to provide specific technical guidance about the use cases and implementation of the technical standards.

• The dialogue is a voluntary group allowing interested experts to join the work at any time.

Item 15: SDMX/DDI dialogue

18

• Only a few NSI's have experience with DDI.• Comments concerned whether Blaise and DDI can be

used together/are compatible.• GSIM will facilitate the integration of SDMX and DDI.• There is a strong need for the ESS to get to a

recommendation for the technical standards to use for micro data exchange in the ESS.

Item 15: SDMX/DDI dialogue

19

• The ESSnet on SDMX phase II runs from March 2011 until September 2012 with IT as co-ordinator + BG, SE, SK.

• The work covers the PC-Axis SDMX integration, how SDMX can be used for micro data handling, harmonization of existing SDMX tools, and capacity building.

• There will also be cooperation between Infostat and DDI Alliance members to work on DDI and SDMX (WP3).

• Improvements of the SDMX Reference Infrastructure.• A Handbook is being produced on "How to use SDMX" for

statisticians.• Two Workshops on the use of SDMX for statisticians and IT-

persons organised in June, Rome 2012.• The deployment of the new version of PC-Axis + functionalities

(like mapping) to be further clarified.

Item 16: ESSnet on SDMX

20

• Eurostat has monitored the national metadata systems on an annual basis since 2008.

• The last monitoring covered the use of business process models and the use and need of metadata for process documentation.

• Most NSIs have or are in the process of adopting a business process model. The use of GSBPM is slowly increasing within the ESS.

• There is a widely recognised need for using standard/common structure for documenting process metadata.

• Countries commented that a monitoring every second year is preferable compared to an annual one.

• UNECE mentioned that the 3rd year questionnaire was used also for surveying the metadata systems of non-EU countries.

• We will make a pause in 2012/2013 and re-discuss the future monitoring in the 2013 MWG.

Item 17: Monitoring of the ESS metadata systems

21

• The Eurostat Process Metadata Structure (EPMS) was developed to ensure the documentation of statistical business processes.

• It complements already existing reference metadata and focuses on the GSBPM steps 5 and 6.

• It is implemented in EMIS and currently around 20 files are opened for production at Eurostat.

• Eurostat will make a first proposal for an ESS process metadata structure and link this proposal to upcoming activities like the ESS VIP Validation.

• It has to be ensured that there will be no overlap with GSIM.

Item 18: ESS Process Metadata Structure

22

• The SDMX work structure was broadened with the putting in place of the two SDMX working groups (the Technical Working Group and the Statistical Working Group) involving also national authorities.

• The two working groups deliver more and more results. The first broader result will be the putting in place of the SDMX Global Registry. More needs to be done for improving the SDMX statistical standards.

• On 13/14 September 2012, there will be another SDMX workshop (at the OECD in Paris) concentrating on the SDMX standards and on SDMX implementation cases.

Item 19: SDMX progress report

23

Item 20: Forthcoming events

• The Expert Group on SDMX, 13-14 September 2012, Paris (hosted by OECD): www.oecd.org/std/research/exchangeexpertgroup

• High Level Seminar on Modernising Statistics, 4-5 October, St Petersburg

• Workshop on Strategic Development In business architecture in statistics , 7-8 November 2012, Geneva

• DIME plenary meeting, Autumn 2012• Classifications Working Group, 7-8 November 2012, Luxembourg• Edamis User Group (former GLC), 13-14 November 2012,

Luxembourg• The IT Directors Group, 28-29 November 2012, Luxembourg• The ESSC, 26 September and 15 November 2012• The METIS work session, Spring 2013