Post on 05-Feb-2016
description
transcript
MDG DATA AND METADATA MDG DATA AND METADATA EXCHANGE AMONG NATIONAL EXCHANGE AMONG NATIONAL
AGENCIES ANDAGENCIES ANDWITH INTERNATIONALWITH INTERNATIONAL
ORGANIZATIONSORGANIZATIONS
The Experience of MexicoThe Experience of Mexico
Enrique OrdazEnrique Ordaz19- 21 October, 201119- 21 October, 2011Manila Manila
Statistics and Geographic Information LawStatistics and Geographic Information Law
National Institute of Statistics National Institute of Statistics and Geography, INEGIand Geography, INEGI
AUTONOMY
NATIONAL COUNCILNATIONAL COUNCIL
INEGIINEGI
SUBSYSTEMSUBSYSTEM SUBSYSTEMSUBSYSTEM SUBSYSTEMSUBSYSTEM SUBSYSTEMSUBSYSTEM
Catalog of National Indicators
National Information Programs
Data dissemination
Norms Coordination
Socio DemographicInformation
SpecializedTechnicalCommittee
UE
Working groups UE
Goverment, PublicSecurity and JusticeInformation
Specialied TechnicalCommittees
Workinggropus
UE
UE
Geographic andEnviromentalInformation
SpecializedTechnicalCommittees
Workinggropus
UE
UE
EconomicInformation
SpecializedTechnicalCommittee
Workinggroups
UE
UE
Society
State
The new statistical The new statistical (and geographic information) system(and geographic information) system
MDG Indicators CommitteeMDG Indicators Committee
A Specialized Technical Committee was set up in February 2010 to
coordinate the integration of the Millennium Development Goals
Indicators.
Mexico’s MDG IndicatorsMexico’s MDG Indicators
Lista oficial de los indicadores de la ONUSistema de Información de los Objetivos de Desarrollo del Milenio,
México 2010
Objetivo Metas Indicadores Metas
Indicadores
Total Preexistente ReformuladosPropuestos por México
De Más allá de las Metas del
Milenio Total 22 70 22 80 40 8 12 20
1. Erradicar la pobreza extrema y el hambre
3 9 5 14 8 1 0 5
2. Lograr la enseñanza primaria universal
1 5 3 14 5 0 1 8
3. Promover la igualdad de género y el empodera-miento de la mujer
1 7 1 7 5 1 1 0
4. Reducir la mortalidad de los niños menores de 5 años
1 3 1 6 3 0 0 3
5. Mejorar la salud materna 2 6 2 6 4 2 0 0
6. Combatir el VIH/SIDA, el paludismo y otras enfermedades
3 13 4 15 4 1 7 3
7. Garantizar la sostenibilidad del medio ambiente
4 12 4 13 8 3 1 1
8. Fomentar una alianza mundial para el desarrollo
6 16 1 3 3 0 0 0
General schedule of the work programGeneral schedule of the work program
Activities
2010 2011Enero Febrero Marzo Abril Mayo Junio Julio Agosto Septiem-
breOctubre Noviem-
breDiciem-
breEnero Febrero
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
•Committee Meetings
•Data integration by working groups
MDG System
1st Phase
2nd Phase
3rd Phase
•Drafting of the 2010 country report
1.Poverty, income and nutrition
Working groups
2. Poverty, employment
3. Education and gender equality
4. Maternal and child mortality, rep. health
5. VIH/AIDS, malaria, other diseases
6. Environment and natural resources
7. Improved water access, and sanitation
8. Global partnerships for development
Ministries responsible
for reviewing and updating
metadata; updating
basic data and indicators
INEGI:technical assistance; data integration and development and management of the website.
General Coordination: Chrístel Rosales (DGPAE)Agreements follow up: César Garcés (CONAPO)
CoordinationCoordination
Process for updating the MDG systemProcess for updating the MDG system
oceso de actualización Sistema ODMoceso de actualización Sistema ODM
ActivitiesActivities
1. Prepare manuals of procedure and rules.
2. Create an FTP site for data exchange.
3. Backing up time series and metadata from the FTP site.
4. Classify, analyze and assess the data and metadata to identify possible inconsistencies.
5. Ask for clarifications from each responsible agency when problems are found in the data.
6. Update the MDG system.
Activities for the operation of the project by INEGI
Form to incorporate data and indicators in Form to incorporate data and indicators in the MDG systemthe MDG system
Metadata formMetadata form
Concept
Name of the indicator
Definition
Algorithm
Meaning of acronyms
Source of primary data
Geographic coverage
Frequency
Updating date
Name of agency responsible for the indicator
Importance and usefulness of the indicator
International reference
Remarks
Procedure to compile and disseminate Procedure to compile and disseminate the information the information
1. The ftp://200.23.8.226/ site contains the following information: Formal the creation of the MDG Technical Committee.
MDG 2006 Manual; draft of the 2009 manual; Report from de Secretary General.
Mexico’s reports from 2005, 2006. Relationship of responsible agencies, indicators and data
sources. Terms of reference of the MDG Technical Committee. Dossiers for each one of the working groups. Each dossier
contains: Indicators’ metadata and the rules for updating it on line. Raw statistics produced or integrated by INEGI for updating
the indicators. A document with the method of calculation for each
indicator
Procedure to compile and disseminate Procedure to compile and disseminate the information the information
2. Within each working group a person is designated to calculate the indicator.
3. INEGI provides access to FTP://200.23.8.226, with different rights.
Procedure to compile and disseminate Procedure to compile and disseminate the information the information
4. Once the work has been finished and authorized, the designated person will up load on the FTP site all the statistical series of raw data to calculate the indicator, as well as the series of the indicator itself, with the corresponding metadata.
5. INEGI reviews and assesses the data and the indicators, and asks for clarifications.
6. Once the data are cleared they are put on an internal server for final review by the Technical Committee.
7. Data are published on the website.
Weekly reportWeekly report
ObservationsObservations
1. Differences in name of indicators: system vs. working groups.
2. Differences or missing metadata.
3. Lack of URL links in the metadata.
4. Acronyms not explained.
5. Inconsistencies in the raw data
6. Differences between the data produced by INEGI and the data published by the government agencies.
7. Unexplained breaks in time series of indicators or the raw data.
Typology of comments made by INEGI on the data produced bay other agencies
Moving MDG to SDMXMoving MDG to SDMX
• We used a preliminary version of the DSD for MDG and we made the following changes and we finally have a NEW DSD for MDG
• It was very useful, but we had to make the following changes in order to adapt it to our information and tools. So we have a NEW DSD for MDG:
– We changed the encoding XML ISO-8859-1 per utf-8 because our tools use encoding = "utf-8“
– The XML namespaces are generated directly from our Web Service for each type of message. So it can be seen that some namespaces (i.e. the required for data messages as "compact", "generic", etc.) do not appear in the DSD proposed by INEGI, but on the other side, it is a new one that is the xmlns: registry that is required in a DSD.
• 34 MDG indicators.• We have adjusted some code lists: 9 were
used, of which 6 are consistent with the UNSD and 3 were included by the INEGI.
• The codes are alphabetically sorted in Spanish.– Two age groups were added because of the
particularities of the indicators presented by Mexico.
– One class added: ND Not determined
Moving MDG to SDMXMoving MDG to SDMX
Progress in the conversion of MDGs to Progress in the conversion of MDGs to
SDMXSDMX
The names of the catalogs were changed, removing _SDMX or _ODM, because by recommendation the identification should be as generic as possible so they can be used by other flows. ANTES AHORA
CL_UNIT_MULT_SDMX CL_UNIT_MULT
CL_NATURE_MDG CL_NATURE
CL_UNIT_MDG CL_UNIT_MEASURE
CL_SOURCE_TYPE_MDG CL_SOURCE_TYPE
CL_LOCATION_MDG CL_LOCATION
CL_AGE_GROUP_MDG CL_AGE_GROUP
CL_SEX_MDG CL_SEX
CL_SERIES_MDG CL_SERIES
CL_REF_AREA_MDG CL_REF_AREA
CL_FREQ_MDG CL_FREQ
CL_UNIT_MULT_SDMX CL_UNIT_MULT
CL_NATURE_MDG CL_NATURE
CL_UNIT_MDG CL_UNIT_MEASURE
CL_SOURCE_TYPE_MDG CL_SOURCE_TYPE
Progress in the conversion of MDG to SDMXProgress in the conversion of MDG to SDMX
• Added ConceptScheme CS_MDG– This component (is required for SDMX 2.0), it contains a list of concepts
applicable to the DSD• Key Family MDG was changed to DSD_MDG because it is recommended
that the nomenclature of the id is based on the first initials of the type of device (DSD DataStructure, Codelist CL, Scheme Concept CS, etc) followed by the identifier.
• The following 3 attributes changed from observation to sets level, because for Mexico, in the retrieval systems and original databases, these features are specified for each series. This change produces smaller SDMX files .
– UNIT_MULT– TIME_DETAIL– FOOTNOTES
• The next three code lists were updated because the INEGI information could not be classified with the original code lists:
– CL_FREQUENCY,– CL_AGE_GROUP – CL_UNIT_MEASURE
UnitedUnited Nations Statistics Division DSD for Nations Statistics Division DSD for
the MDGthe MDG
• DSD STRUCTURE CODELIST (CL) APPLICATION
DIMENSIONS
1. Time period Yes
2. Frequency CL_FREQ_MDG Yes- CL Supplemented3. Series CL_SERIES_MDG Yes
4. Location CL_LOCATION_MDG Yes
5. Sex CL_SEX_MDG Yes
6. Age group CL_AGE_GROUP_MDG Yes- CL Supplemented
7. Reference area CL_REF_AREA_MDG Yes
8. Units of measurement CL_UNIT_MDG Yes- CL Supplemented9. Source Type CL_SOURCE_TYPE_MDG Yes
ATTRIBUTES
1. Unit multiplier CL_UNIT_MULT_SDMX Yes
2. Time period details Yes
3. Nature of data points CL_NATURE Yes
4. Source Detail Yes
5. Footnotes Yes
CODE LIST FOR UNITS OF MEASUREMENTCODE LIST FOR UNITS OF MEASUREMENTCODE DESCRIPTION SOURCE APPLICATION
1 Not applicable ONU Not used2 Deaths of children under five per thousand live births INEGI-New Used3 Deaths of children under one year per thousand live births INEGI-New Used4 USD in end-2006 net present value terms ONU Not used5 USD ONU Not used6 Children per thousand women INEGI-New Used7 Number ONU Used8 Kilograms per person INEGI-New Used9 Kg oil equivalent per USD1,000 constant 2005 PPP GDP ONU Not used
10 Square kilometers ONU Not used11 Local currency ONU Not used12 Local currency per USD (PPP) ONU Not used13 Women ONU Not used14 Women for men INEGI-New Used15 Live Births ONU Not used16 Population ONU Not used17 Per hundred thousand INEGI-New Used18 Percent ONU Used19 Per 1 USD GDP (PPP) ONU Not used20 Metric tons ONU Used
Nota: For the Mexico’s DSD of 34 MDG indicators, 9 classes were used (in purple), of which 3 are consistent with the Catalogue of the UN and 6 were included by the INEGI. About the codes, were used consecutive numbers, arranged sorted alphabetically terms in Spanish
CODE LIST FOR AGE GROUPSCODE DESCRIPTION SOURCE APPLICATION
Z Not applicable ONU Yes
1 Under 1 year olds ONU Yes
2 Under 5 year olds ONU Yes
3 6-11 year olds INEGI Yes
4 10-14 year olds ONU Not used
5 14 and more year olds INEGI Yes
6 15-19 year olds ONU Yes
7 15-24 year olds ONU Yes
8 15-49 year olds ONU Yes
9 All age ranges ONU Yes
Note: Two age groups were added because of the particularities of the indicators presented by Mexico.
CODE LIST FOR FREQUENCY
CODE DESCRIPTION SOURCE APPLICATION
A Annual ONU Yes
2A Two-year average ONU Yes
3A Three-year average ONU Not used
S Half-yearly, semester ONU Not used
Q Quarterly ONU Not used
M Monthly ONU Not used
ND Not determined INEGI – NewYes
Note: Added one class to characterize the statistics that have no clearly defined periodicity: ND: Not determined
SDMX – MDGConsulta del DSD desde el Web
Service de INEGIDataFlows de INEGI (Para saber qué flujos disponibles hay y sus correspondientes DSD se
hace primero una consulta dinámica de todos los DataFlows publicados por INEGI)http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Dataflow/ALL/ALL/ALL
(o si ya se sabe el nombre del DataFlow se puede acotar por ejemplo a DF_MDG)http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Dataflow/ALL/DF_MDG/ALL
El nombre del DSD de MDG se puede ver en el resultado de la consulta anterior. Por ejemplo, para el caso de los MDG el nombre es DSD_MDG. Para consultarlo se construye la URL como sigue:
DataStructurehttp://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/DataStructure/ALL/DSD_MDG/ALL
DataStructure con artefactos de referencias (codelist, consept scheme)http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/DataStructure/ALL/DSD_MDG/ALL?references=shallow
CODELISTS (Ejemplo para consultar dinámicamente el código CL_FREQ, los nombres de los códigos los sabemos en el DSD de la consulta anterior)
http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Codelist/IAEG/CL_FREQ/ALL
SDMX – MDGConsulta de Datos desde el Web Service de INEGI en diferentes
formatosDATOS del Flujo ODM INEGIDatos xml (Se pueden visualizar en el Explorer o en cualquier visualizador como
XML-Marker)http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?
Key=ALL&format=compact
Datos xml filtrados, Ejemplo (Trae exclusivamente el indicador de MEXICO cuya SERIE es SL_EMP_TOTL)
KEY = [FREQ].[REF_AREA].[SERIES].[SEX].[AGE_GROUP].[LOCATION].[SOURCE_TYPE].[UNIT]
http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?Key=.MEX.SL_EMP_TOTL.....&format=compact
Datos csv (Archivo separado por comas, reduciendo mas del 50% de su tamaño xml, y se puede visualizar en varias herramientas como Excel, ó llenar directamente una tabla de BD)
http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?Key=ALL&alt=csv
Datos chart (Trae una imagen, este es el mismo ejemplo anterior de datos filtrados, que trae sólo el indicador de MEXICO cuya SERIE es SL_EMP_TOTL, para Total, Hombres y Mujeres, en una imagen)
http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?Key=.MEX.SL_EMP_TOTL.....&format=compact&alt=chart
Datos json y jsonp (Para utilizar los datos del flujo en aplicaciones WEB de terceros como Facebook, Twitter, etc. -OPEN DATA-)
http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?Key=ALL&format=compact&alt=jsonhttp://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?
Key=ALL&format=compact&alt=json&callback=jsonp
SDMX – MDGCAMBIOS EN EL DSD
• Dynamic chart sourced by a SDMX flow
Proposed Timeline for Conversion to SDMX the MDG indicators
Actividades A2
A3
A4
A5
S1
S2
S3
S4
O1
O2
O3
O4
Area responsible
1. Analysis of the DSD - MDG provided by the UN and its applicability to integrated indicators in the Mexican project
* * DGAI – DGAII
2. Meeting with conceptual and database staffs to present the structure of DSD (dimensions and attributes) and catalogs, in order to analyze the concepts used in the DSD and to define from where will be taken
* DGAI – DGAII
3. Establishing the equivalence between the dimensions, attributes and catalogs of DSD vs Project database
* * DGAII
4. Analysis of the database for the transformation process
* * DGAI – DGAII
5. Amendment proposed to the DSD * DGAI
Proposed Timeline for Conversion to SDMX the MDG indicators
Actividades A2
A3
A4
A5
S1
S2
S3
S4
O1
O2
O3
O4
Area responsible
6. Generating of a SQLServer scheme database and migrate data from Access, or agreement the access if there is a copy
* * DGAI-DGAII
7. Generating a SQL query to the database to obtain the specific information for the required flow
* DGAI
8. Loading the modified DSD and the query to the Mapping Assistant and make appropriate correlations, saving it with its stream identifier
* * DGAI
9. Run the flow and validate the data against the original data file, and make corrections where necessary
* * * * DGAI - DGAII
10. Plenary meeting with conceptual staff in order to present them the results
* * * DGAI
11. Developing a plenary meeting with conceptual staff in order to present the results.
* DGAI - DGAII
12. Sending the flow to the UN for validation * DGIAI
MDG DATA EXCHANGE AMONG MDG DATA EXCHANGE AMONG NATIONAL AGENCIES ANDNATIONAL AGENCIES AND
WITH INTERNATIONALWITH INTERNATIONALORGANIZATIONSORGANIZATIONS
The Experience of MexicoThe Experience of Mexico
Enrique OrdazEnrique Ordaz19- 21 October, 201119- 21 October, 2011Manila Manila