MDG DATA AND METADATA EXCHANGE AMONG NATIONAL AGENCIES AND WITH INTERNATIONAL ORGANIZATIONS The Experience of Mexico Enrique Ordaz 19- 21 October, 2011.

Presentación del tema: "MDG DATA AND METADATA EXCHANGE AMONG NATIONAL AGENCIES AND WITH INTERNATIONAL ORGANIZATIONS The Experience of Mexico Enrique Ordaz 19- 21 October, 2011."— Transcripción de la presentación:

1 MDG DATA AND METADATA EXCHANGE AMONG NATIONAL AGENCIES AND WITH INTERNATIONAL ORGANIZATIONS The Experience of Mexico Enrique Ordaz 19- 21 October, 2011 Manila

2 Statistics and Geographic Information Law

3 National Institute of Statistics and Geography, INEGI AUTONOMY

4 NATIONAL COUNCIL INEGI SUBSYSTEMSUBSYSTEM SUBSYSTEM SUBSYSTEM Catalog of National Indicators National Information Programs Data dissemination NormsCoordination Socio Demographic Information Specialized Technical Committee UE Working groups UE Goverment, Public Security and Justice Information Specialied Technical Committees Working gropus UE Geographic and Enviromental Information Specialized Technical Committees Working gropus UE Economic Information Specialized Technical Committee Working groups UE Society State The new statistical (and geographic information) system

5 MDG Indicators Committee A Specialized Technical Committee was set up in February 2010 to coordinate the integration of the Millennium Development Goals Indicators.

6 Mexicos MDG Indicators Lista oficial de los indicadores de la ONU Sistema de Información de los Objetivos de Desarrollo del Milenio, México 2010 ObjetivoMetasIndicadoresMetas Indicadores TotalPreexistenteReformulados Propuestos por México De Más allá de las Metas del Milenio Total 227022804081220 1.Erradicar la pobreza extrema y el hambre 395148105 2.Lograr la enseñanza primaria universal 153145018 3.Promover la igualdad de género y el empodera- miento de la mujer 17175110 4.Reducir la mortalidad de los niños menores de 5 años 13163003 5.Mejorar la salud materna26264200 6.Combatir el VIH/SIDA, el paludismo y otras enfermedades 3134154173 7.Garantizar la sostenibilidad del medio ambiente 4124138311 8.Fomentar una alianza mundial para el desarrollo 616133000

7 General schedule of the work program Activities 20102011 EneroFebreroMarzoAbrilMayoJunioJulioAgostoSeptiem- bre OctubreNoviem- bre Diciem- bre EneroFebrero 12341234123412341234123412341234123412341234123412341234 Committee Meetings Data integration by working groups MDG System 1 st Phase 2 nd Phase 3 rd Phase Drafting of the 2010 country report

8 1.Poverty, income and nutrition Working groups 2. Poverty, employment 3. Education and gender equality 4. Maternal and child mortality, rep. health 5. VIH/AIDS, malaria, other diseases 6. Environment and natural resources 7. Improved water access, and sanitation 8. Global partnerships for development Ministries responsible for reviewing and updating metadata; updating basic data and indicators INEGI: technical assistance; data integration and development and management of the website. General Coordination: Chrístel Rosales (DGPAE) Agreements follow up: César Garcés (CONAPO) Coordination

9 Process for updating the MDG system oceso de actualización Sistema ODM

10 Activities 1.Prepare manuals of procedure and rules. 2.Create an FTP site for data exchange. 3.Backing up time series and metadata from the FTP site. 4.Classify, analyze and assess the data and metadata to identify possible inconsistencies. 5.Ask for clarifications from each responsible agency when problems are found in the data. 6.Update the MDG system. Activities for the operation of the project by INEGI

11 Form to incorporate data and indicators in the MDG system

12 Metadata form Concept Name of the indicator Definition Algorithm Meaning of acronyms Source of primary data Geographic coverage Frequency Updating date Name of agency responsible for the indicator Importance and usefulness of the indicator International reference Remarks

13 Procedure to compile and disseminate the information 1.The ftp://200.23.8.226/ site contains the following information:ftp://200.23.8.226/ Formal the creation of the MDG Technical Committee. MDG 2006 Manual; draft of the 2009 manual; Report from de Secretary General. Mexicos reports from 2005, 2006. Relationship of responsible agencies, indicators and data sources. Terms of reference of the MDG Technical Committee. Dossiers for each one of the working groups. Each dossier contains: Indicators metadata and the rules for updating it on line. Raw statistics produced or integrated by INEGI for updating the indicators. A document with the method of calculation for each indicator

14 Procedure to compile and disseminate the information 2.Within each working group a person is designated to calculate the indicator. 3.INEGI provides access to FTP://200.23.8.226, with different rights.

15 Procedure to compile and disseminate the information 4.Once the work has been finished and authorized, the designated person will up load on the FTP site all the statistical series of raw data to calculate the indicator, as well as the series of the indicator itself, with the corresponding metadata. 5.INEGI reviews and assesses the data and the indicators, and asks for clarifications. 6.Once the data are cleared they are put on an internal server for final review by the Technical Committee. 7.Data are published on the website.

16 Weekly report

17 Observations 1.Differences in name of indicators: system vs. working groups. 2.Differences or missing metadata. 3.Lack of URL links in the metadata. 4.Acronyms not explained. 5.Inconsistencies in the raw data 6.Differences between the data produced by INEGI and the data published by the government agencies. 7.Unexplained breaks in time series of indicators or the raw data. Typology of comments made by INEGI on the data produced bay other agencies

18

19 Moving MDG to SDMX We used a preliminary version of the DSD for MDG and we made the following changes and we finally have a NEW DSD for MDGDSD for MDG NEW DSD for MDG It was very useful, but we had to make the following changes in order to adapt it to our information and tools. So we have a NEW DSD for MDG: – We changed the encoding XML ISO-8859-1 per utf-8 because our tools use encoding = "utf-8 – The XML namespaces are generated directly from our Web Service for each type of message. So it can be seen that some namespaces (i.e. the required for data messages as "compact", "generic", etc.) do not appear in the DSD proposed by INEGI, but on the other side, it is a new one that is the xmlns: registry that is required in a DSD.

20 34 MDG indicators. We have adjusted some code lists: 9 were used, of which 6 are consistent with the UNSD and 3 were included by the INEGI. The codes are alphabetically sorted in Spanish. – Two age groups were added because of the particularities of the indicators presented by Mexico. – One class added: ND Not determined Moving MDG to SDMX

21 Progress in the conversion of MDGs to SDMX The names of the catalogs were changed, removing _SDMX or _ODM, because by recommendation the identification should be as generic as possible so they can be used by other flows. ANTESAHORA CL_UNIT_MULT_SDMXCL_UNIT_MULT CL_NATURE_MDGCL_NATURE CL_UNIT_MDGCL_UNIT_MEASURE CL_SOURCE_TYPE_MDGCL_SOURCE_TYPE CL_LOCATION_MDGCL_LOCATION CL_AGE_GROUP_MDGCL_AGE_GROUP CL_SEX_MDGCL_SEX CL_SERIES_MDGCL_SERIES CL_REF_AREA_MDGCL_REF_AREA CL_FREQ_MDGCL_FREQ CL_UNIT_MULT_SDMXCL_UNIT_MULT CL_NATURE_MDGCL_NATURE CL_UNIT_MDGCL_UNIT_MEASURE CL_SOURCE_TYPE_MDGCL_SOURCE_TYPE

22 Progress in the conversion of MDG to SDMX Added ConceptScheme CS_MDG – This component (is required for SDMX 2.0), it contains a list of concepts applicable to the DSD Key Family MDG was changed to DSD_MDG because it is recommended that the nomenclature of the id is based on the first initials of the type of device (DSD DataStructure, Codelist CL, Scheme Concept CS, etc) followed by the identifier. The following 3 attributes changed from observation to sets level, because for Mexico, in the retrieval systems and original databases, these features are specified for each series. This change produces smaller SDMX files. – UNIT_MULT – TIME_DETAIL – FOOTNOTES The next three code lists were updated because the INEGI information could not be classified with the original code lists: – CL_FREQUENCY, – CL_AGE_GROUP – CL_UNIT_MEASURE

23 United Nations Statistics Division DSD for the MDG DSD STRUCTURECODELIST (CL)APPLICATION DIMENSIONS 1.Time periodYes 2. FrequencyCL_FREQ_MDG Yes- CL Supplemented 3. SeriesCL_SERIES_MDGYes 4. LocationCL_LOCATION_MDGYes 5. SexCL_SEX_MDGYes 6. Age groupCL_AGE_GROUP_MDG Yes- CL Supplemented 7. Reference areaCL_REF_AREA_MDGYes 8. Units of measurementCL_UNIT_MDG Yes- CL Supplemented 9. Source TypeCL_SOURCE_TYPE_MDGYes ATTRIBUTES 1. Unit multiplierCL_UNIT_MULT_SDMXYes 2. Time period detailsYes 3. Nature of data pointsCL_NATUREYes 4. Source Detail Yes 5. FootnotesYes

24 CODE LIST FOR UNITS OF MEASUREMENT CODEDESCRIPTIONSOURCEAPPLICATION 1 Not applicableONUNot used 2 Deaths of children under five per thousand live birthsINEGI-NewUsed 3 Deaths of children under one year per thousand live birthsINEGI-NewUsed 4 USD in end-2006 net present value termsONUNot used 5 USDONUNot used 6 Children per thousand womenINEGI-NewUsed 7 NumberONUUsed 8 Kilograms per personINEGI-NewUsed 9 Kg oil equivalent per USD1,000 constant 2005 PPP GDPONUNot used 10 Square kilometersONUNot used 11 Local currencyONUNot used 12 Local currency per USD (PPP)ONUNot used 13 WomenONUNot used 14 Women for menINEGI-NewUsed 15 Live BirthsONUNot used 16 PopulationONUNot used 17 Per hundred thousandINEGI-NewUsed 18 PercentONUUsed 19 Per 1 USD GDP (PPP)ONUNot used 20 Metric tonsONUUsed Nota: For the Mexicos DSD of 34 MDG indicators, 9 classes were used (in purple), of which 3 are consistent with the Catalogue of the UN and 6 were included by the INEGI. About the codes, were used consecutive numbers, arranged sorted alphabetically terms in Spanish

25 CODE LIST FOR AGE GROUPS CODEDESCRIPTIONSOURCEAPPLICATION ZNot applicableONU Yes 1Under 1 year oldsONU Yes 2Under 5 year oldsONU Yes 36-11 year oldsINEGI Yes 410-14 year oldsONU Not used 514 and more year oldsINEGI Yes 615-19 year oldsONU Yes 715-24 year oldsONU Yes 815-49 year oldsONU Yes 9All age rangesONU Yes Note: Two age groups were added because of the particularities of the indicators presented by Mexico.

26 CODE LIST FOR FREQUENCY CODEDESCRIPTIONSOURCEAPPLICATION AAnnualONUYes 2ATwo-year averageONUYes 3AThree-year averageONUNot used SHalf-yearly, semesterONUNot used QQuarterlyONUNot used MMonthlyONUNot used NDNot determinedINEGI – New Yes Note: Added one class to characterize the statistics that have no clearly defined periodicity: ND: Not determined

27 SDMX – MDG Consulta del DSD desde el Web Service de INEGI DataFlows de INEGI (Para saber qué flujos disponibles hay y sus correspondientes DSD se hace primero una consulta dinámica de todos los DataFlows publicados por INEGI) http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Dataflow/ALL/ALL/ALL (o si ya se sabe el nombre del DataFlow se puede acotar por ejemplo a DF_MDG) http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Dataflow/ALL/DF_MDG/ALL El nombre del DSD de MDG se puede ver en el resultado de la consulta anterior. Por ejemplo, para el caso de los MDG el nombre es DSD_MDG. Para consultarlo se construye la URL como sigue: DataStructure http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/DataStructure/ALL/DSD_MDG/ALL DataStructure con artefactos de referencias (codelist, consept scheme) http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/DataStructure/ALL/DSD_MDG/ALL?reference s=shallow CODELISTS (Ejemplo para consultar dinámicamente el código CL_FREQ, los nombres de los códigos los sabemos en el DSD de la consulta anterior) http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Codelist/IAEG/CL_FREQ/ALL

28 SDMX – MDG Consulta de Datos desde el Web Service de INEGI en diferentes formatos DATOS del Flujo ODM INEGI Datos xml (Se pueden visualizar en el Explorer o en cualquier visualizador como XML-Marker) http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?Key=ALL&format=com pact Datos xml filtrados, Ejemplo (Trae exclusivamente el indicador de MEXICO cuya SERIE es SL_EMP_TOTL) KEY = [FREQ].[REF_AREA].[SERIES].[SEX].[AGE_GROUP].[LOCATION].[SOURCE_TYPE].[UNIT] http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?Key=.MEX.SL_EMP_TO TL.....&format=compact Datos csv (Archivo separado por comas, reduciendo mas del 50% de su tamaño xml, y se puede visualizar en varias herramientas como Excel, ó llenar directamente una tabla de BD) http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?Key=ALL&alt=csv Datos chart (Trae una imagen, este es el mismo ejemplo anterior de datos filtrados, que trae sólo el indicador de MEXICO cuya SERIE es SL_EMP_TOTL, para Total, Hombres y Mujeres, en una imagen) http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?Key=.MEX.SL_EMP_TOTL.....&format=co mpact&alt=chart Datos json y jsonp (Para utilizar los datos del flujo en aplicaciones WEB de terceros como Facebook, Twitter, etc. -OPEN DATA-) http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?Key=ALL&format=compact&alt=json http://www.sdmx.snieg.mx/sistemas/sdmx/restsdmx/Data/DF_MDG/INEGI?Key=ALL&format=compact&alt=json&ca llback=jsonp

29 SDMX – MDG CAMBIOS EN EL DSD Dynamic chart sourced by a SDMX flow

30

31

32

33

34 Proposed Timeline for Conversion to SDMX the MDG indicators Actividades A2A2 A3A3 A4A4 A5A5 S1S1 S2S2 S3S3 S4S4 O1O1 O2O2 O3O3 O4O4 Area responsible 1.Analysis of the DSD - MDG provided by the UN and its applicability to integrated indicators in the Mexican project ** DGAI – DGAII 2. Meeting with conceptual and database staffs to present the structure of DSD (dimensions and attributes) and catalogs, in order to analyze the concepts used in the DSD and to define from where will be taken * DGAI – DGAII 3. Establishing the equivalence between the dimensions, attributes and catalogs of DSD vs Project database ** DGAII 4. Analysis of the database for the transformation process ** DGAI – DGAII 5. Amendment proposed to the DSD * DGAI

35 Proposed Timeline for Conversion to SDMX the MDG indicators Actividades A2A2 A3A3 A4A4 A5A5 S1S1 S2S2 S3S3 S4S4 O1O1 O2O2 O3O3 O4O4 Area responsible 6.Generating of a SQLServer scheme database and migrate data from Access, or agreement the access if there is a copy ** DGAI-DGAII 7.Generating a SQL query to the database to obtain the specific information for the required flow * DGAI 8.Loading the modified DSD and the query to the Mapping Assistant and make appropriate correlations, saving it with its stream identifier ** DGAI 9.Run the flow and validate the data against the original data file, and make corrections where necessary **** DGAI - DGAII 10.Plenary meeting with conceptual staff in order to present them the results *** DGAI 11. Developing a plenary meeting with conceptual staff in order to present the results. * DGAI - DGAII 12.Sending the flow to the UN for validation * DGIAI

36 MDG DATA EXCHANGE AMONG NATIONAL AGENCIES AND WITH INTERNATIONAL ORGANIZATIONS The Experience of Mexico Enrique Ordaz 19- 21 October, 2011 Manila