Sdmx at australian bureau of statistics

SDMX @ ABS: SDMX @ ABS: Why and How we use Why and How we use

SDMXSDMX

Graeme OakleyGraeme OakleyAustralian Bureau of Australian Bureau of

StatisticsStatistics

"Towards Implementation of SDMX - Jan 9-11, 2007"

Content of PresentationContent of Presentation

Introduction - objectives, overview Introduction - objectives, overview of ABSof ABS

Principles relevant to ABS data and Principles relevant to ABS data and metadata dissemination metadata dissemination

Case studies - one national, one Case studies - one national, one internationalinternational

Vision, Lessons, IssuesVision, Lessons, Issues



Objectives of PresentationObjectives of Presentation

Why SDMX fits with ABS philosophy Why SDMX fits with ABS philosophy concerning principles and standardsconcerning principles and standards

Tell our stories about a national and Tell our stories about a national and international experience using SDMXinternational experience using SDMX

Discuss the future vision, concerns Discuss the future vision, concerns and lessonsand lessons

Australian Bureau of Australian Bureau of StatisticsStatisticsAustralia's national statistical agency, Australia's national statistical agency, statistical leadership role in National statistical leadership role in National Statistical ServiceStatistical Service

ABS MissionABS Mission".. assist and encourage informed decision-".. assist and encourage informed decision-making, research and discussion .."making, research and discussion .."

A broad range of statistics on Australian A broad range of statistics on Australian societysocietyEconomic, Demographic, Social, Health, Economic, Demographic, Social, Health, Tourism, Culture and Leisure, Tourism, Culture and Leisure, Environment, ...Environment, ...

Central and 8 Regional offices, about 2750 Central and 8 Regional offices, about 2750 staff (FTE) plus 800 interviewers staff (FTE) plus 800 interviewers


ABS PrinciplesABS Principles

RelevanceRelevance: good planning with a good : good planning with a good understanding of current and future understanding of current and future needs of users; needs of users; timely and relatable timely and relatable statisticsstatistics

IntegrityIntegrity: analysis and interpretation is : analysis and interpretation is objective; always publish our statistics; objective; always publish our statistics; explain and informexplain and inform; methodologies and ; methodologies and approaches open to scrutinyapproaches open to scrutiny

Access for allAccess for all: strive for accurate and : strive for accurate and widespread media coverage; widespread media coverage; access access through range of products and servicesthrough range of products and services


ABS Principles (ctd)ABS Principles (ctd)

ProfessionalismProfessionalism: exercise highest : exercise highest professional standards; staff highly professional standards; staff highly trained; trained; development of statistical development of statistical standardsstandards; open about the ; open about the quality of our quality of our statisticsstatistics

Trust of providersTrust of providers: encouraged to provide : encouraged to provide accurate information; accurate information; data is strictly data is strictly protectedprotected; explain why information is ; explain why information is collected; every ABS officer required to collected; every ABS officer required to give an undertaking of secrecygive an undertaking of secrecy

Source: ABS Corporate PlanSource: ABS Corporate Plan


ABS - ClientsABS - Clients

Australia's national statistical agency Australia's national statistical agency serving:serving:Australian GovernmentAustralian GovernmentState/Territory GovernmentsState/Territory GovernmentsLocal GovernmentsLocal GovernmentsInternational statistical organisationsInternational statistical organisationsBusiness communityBusiness communityEducation, religious, welfare, etc Education, religious, welfare, etc organisationsorganisations

MediaMediaIndividual members of societyIndividual members of society"Towards Implementation of SDMX - Jan 9-11, 2007"

Principles and Standards Principles and Standards ApproachApproachData ManagementData Management - data is visible, - data is visible, accessible, understandable, reliable, accessible, understandable, reliable, relatable, presentable relatable, presentable

MetadataMetadata - definitions (understanding), - definitions (understanding), discovery, quality ('fitness for purpose') discovery, quality ('fitness for purpose')

ABS Enterprise ArchitectureABS Enterprise Architecture - follow - follow mainstream standards, collaborate & mainstream standards, collaborate & partner where effective, promote partner where effective, promote development of low complexity & development of low complexity & maintainable solutions, remove maintainable solutions, remove dependence on vendor formats, extensible dependence on vendor formats, extensible and scalable systems, use emerging and scalable systems, use emerging technology as appropriatetechnology as appropriate


Standards (ctd)Standards (ctd)


DisseminationDissemination - - all publishable data in ABS Information all publishable data in ABS Information Warehouse Warehouse

metadata associated with products metadata associated with products web site is prime dissemination channel web site is prime dissemination channel clients can 'pull' data for free clients can 'pull' data for free data stored once but published in many data stored once but published in many common formatscommon formats

StatisticalStatistical - use international standards - use international standards eg ICD, SITC; and frameworks eg SNAeg ICD, SITC; and frameworks eg SNA

Case StudiesCase StudiesInternationalInternational - NAWWE with OECD - NAWWE with OECD

(Note: ABS responds to many (Note: ABS responds to many information requests from international information requests from international organisations, in variety of formats - we organisations, in variety of formats - we are seeking an efficient solution that are seeking an efficient solution that aligns with our principles and standards)aligns with our principles and standards)

NationalNational - Reserve Bank of Australia - Reserve Bank of Australia (RBA)(RBA)

[A large, sophisticated and important user [A large, sophisticated and important user of ABS data. There are a number of other of ABS data. There are a number of other users of this type eg Treasury.]users of this type eg Treasury.]"Towards Implementation of SDMX - Jan 9-11, 2007"

International - OECD NAWWEInternational - OECD NAWWEInvolvement in pilot project since Involvement in pilot project since

2002/032002/03Tasks include:Tasks include:

Set up metadata and load National Set up metadata and load National Accounts data to ABS Information Accounts data to ABS Information Warehouse to suit agreed NAWWE Data Warehouse to suit agreed NAWWE Data Structure DefinitionStructure Definition

Extract from ABSIW and transform to Extract from ABSIW and transform to agreed SDMX message structure (XML agreed SDMX message structure (XML format)format)

Place on ABS website (recently Place on ABS website (recently decommissioned web page for original decommissioned web page for original case study and yet to put up one for case study and yet to put up one for current study)current study)


OECD

FAME Time Series DB

ABS Information Warehouse

ABS Statistical Products

EXCEL File

Hand entered

Current Process


OECD

FAME Time Series DB

ABS Information Warehouse

ABS Statistical Products

Time Series XML

Transform to SDMX

Publish to www.abs.gov.a

u

Future Process


ABS ExperienceABS Experience

Changed business process requiredChanged business process requiredAlthough supporting the initiative, ABS Although supporting the initiative, ABS

National Accounts group could not National Accounts group could not always provide resources to do always provide resources to do necessary setup worknecessary setup work

Long periods on inactivity whilst OECD Long periods on inactivity whilst OECD worked on new Data Structure Definitionworked on new Data Structure Definition

Experimental SDMX V1 files were Experimental SDMX V1 files were delivereddelivered

Now have SDMX V2 DSD to understand Now have SDMX V2 DSD to understand and implementand implement


National - RBANational - RBA


Central Bank for Australia (RBA) - Central Bank for Australia (RBA) - receives mainly time series data from receives mainly time series data from many sourcesmany sources

Many formats and different Many formats and different frequenciesfrequencies

Previously maintained their time Previously maintained their time series data in an EXCEL based series data in an EXCEL based spreadsheet systemspreadsheet system

In past year, developed new system In past year, developed new system to centralise time series data storage to centralise time series data storage and retrieval, based on SQL serverand retrieval, based on SQL server

National - RBA (ctd)National - RBA (ctd)

ABS provided detailed Labour Force ABS provided detailed Labour Force data in SDMX V1 format as a trial (RBA data in SDMX V1 format as a trial (RBA currently takes LFS data in a large currently takes LFS data in a large number of EXCEL files and some number of EXCEL files and some multidimensional cubes)multidimensional cubes)

The RBA's new system has been The RBA's new system has been demonstrated to internal business demonstrated to internal business areas, and acceptedareas, and accepted

RBA want to extend use of SDMX to RBA want to extend use of SDMX to other ABS series and improve the other ABS series and improve the timelinesstimeliness


ABS RBALabour Force data in SDMX

format

SQL Server DB holding time series data

XSLT

SQL Server Integration Services (SSIS)

Email with file attached

What RBA do with SDMX files


RBA economists

What RBA see as benefits?What RBA see as benefits?

Efficient data transfer - reduces need for Efficient data transfer - reduces need for clerical intervention when dealing with clerical intervention when dealing with spreadsheets, can easily manipulate with spreadsheets, can easily manipulate with programsprograms

Metadata rich - SDMX allows more Metadata rich - SDMX allows more metadata to be conveyed compared to metadata to be conveyed compared to spreadsheetsspreadsheets

Able to handle changes in time series Able to handle changes in time series metadata and data more easily - can metadata and data more easily - can programmatically check metadata for programmatically check metadata for changeschanges


What does RBA want ABS to What does RBA want ABS to change?change?


Nothing related to the transfer format - Nothing related to the transfer format - SDMXSDMX

Want the SDMX files on ABS web site Want the SDMX files on ABS web site so can download at 11.30am (embargo so can download at 11.30am (embargo time) rather than wait for emailtime) rather than wait for email

Want ABS to extend use of SDMX to Want ABS to extend use of SDMX to other statistical seriesother statistical series

Additional information related to Additional information related to identification of discontinued series, identification of discontinued series, code setscode sets

VisionVision


Vision - framework and standard that Vision - framework and standard that ABS can use for all our international ABS can use for all our international reporting; and also to use with large reporting; and also to use with large major national users of ABS data.major national users of ABS data.

SDMX is suitable as the primary SDMX is suitable as the primary

dissemination vehicle for machine dissemination vehicle for machine readable aggregate and time series readable aggregate and time series datasets for both national and datasets for both national and international clients. The SDMX format international clients. The SDMX format files will become ABS data products held files will become ABS data products held on the web site for free access.on the web site for free access.

Lessons Lessons

SDMX has the potential to provide SDMX has the potential to provide integrated data exchange across integrated data exchange across organisational boundaries. Challenge is organisational boundaries. Challenge is not the SDMX 'technology', but dealing not the SDMX 'technology', but dealing with 'meaning' and 'mapping' issueswith 'meaning' and 'mapping' issues

Recognise costs involved in Recognise costs involved in implementation, especially mapping local implementation, especially mapping local metadata and data structures to metadata and data structures to international requirement in each subject international requirement in each subject domain, and the importance of engaging domain, and the importance of engaging local subject matter expertslocal subject matter experts


IssuesIssues


Use SDMX for wide range of users, from Use SDMX for wide range of users, from technically competent to small data technically competent to small data integrator. A short shallow learning curve integrator. A short shallow learning curve with plenty of support, such as easy with plenty of support, such as easy documentation, examples, tutorials, and documentation, examples, tutorials, and toolstools

Simple time series identifier, in addition to Simple time series identifier, in addition to the Data Structure Definition - eg local the Data Structure Definition - eg local identifier as part of exchange - the key identifier as part of exchange - the key back to internal data holdings; existing back to internal data holdings; existing international 'code' for each series used in international 'code' for each series used in current exchanges - doesn't vary from current exchanges - doesn't vary from country to country.country to country.

ConclusionConclusion


Move to SDMX V2, recognising need to Move to SDMX V2, recognising need to keep back-end systems independent of keep back-end systems independent of version changes.version changes.

Importance of establishing business Importance of establishing business benefit of SDMX - it is not just a benefit of SDMX - it is not just a technology.technology.

Ideal of a common framework that would Ideal of a common framework that would guide each statistical domain to guide each statistical domain to determine their exchange requirements determine their exchange requirements and definitions.and definitions.

Need for good registration processes Need for good registration processes around all aspects of SDMX, and around all aspects of SDMX, and involvement of NSO's in governance.involvement of NSO's in governance.

QUESTIONS ANDDISCUSSION


Date post:	16-Nov-2014
Category:	Documents
Upload:	vinicius-silva
View:	356 times
Download:	2 times

Sdmx at australian bureau of statistics

Documents