Date post: | 13-Jan-2016 |
Category: |
Documents |
Upload: | jessie-harrison |
View: | 217 times |
Download: | 1 times |
Eurostat
Quality indicators for statistics based on
multiple sources
Mihaela Agafiţei, Fabrice Gras, Wim Kloek, Sorina VâjuEurostat, European Commission
Eurostat
Content
1. Introduction
2. Quality of statistics – general discussion
3. Output quality assessment – input and process
4. Direct output quality assessment
5. Conclusions
2/11Q2014 – Vienna – 5th of June, 2014
Session No 32 - Statistics beyond survey and administrative dataQuality indicators for statistics based on multiple sources
Mihaela Agafiţei, Fabrice Gras, Wim Kloek, Sorina Vâju(Eurostat, European Commission)
Eurostat
Challenges
• Reduce response burden
• Reduce cost of raw data collection
• Increase ability to face new demands
• Increase ability to produce more detailed statistics
Increase use of administrative data sources
•Direct use•Use in sampling frame•Auxiliary information•Calibration
Output quality
assessment
• Can consider the integration effect?
• Can consider the variety of statistical approaches
• Can advantages be offset by possible decreases in the quality?
1. Introduction
3/11Quality indicators for statistics based on multiple sources
Mihaela Agafiţei, Fabrice Gras, Wim Kloek, Sorina Vâju(Eurostat, European Commission)
Q2014 – Vienna – 5th of June, 2014Session No 32 - Statistics beyond survey and administrative data
Eurostat
2. Quality of statistics – general discussion
Input
Quality of raw data
Whether and how a given data source can be used on a regular basis to produce statistics
Process
Whether final data is “real”
Magnitude of errors introduced in processing stage
Analyse of statistical process
Output
User easy to understand information on the quality of the final data
4/11Quality indicators for statistics based on multiple sources
Mihaela Agafiţei, Fabrice Gras, Wim Kloek, Sorina Vâju(Eurostat, European Commission)
Q2014 – Vienna – 5th of June, 2014Session No 32 - Statistics beyond survey and administrative data
Eurostat
2. Quality of statistics – general discussion
The ESS Code of Practice
The ESS Standard for Quality Reports
ESS Handbook for Quality Reports
RelevanceAccuracy
&Reliability
Timeliness &
Punctuality
Coherence &Comparability
Accessibility &
Clarity
5/11Quality indicators for statistics based on multiple sources
Mihaela Agafiţei, Fabrice Gras, Wim Kloek, Sorina Vâju(Eurostat, European Commission)
Q2014 – Vienna – 5th of June, 2014Session No 32 - Statistics beyond survey and administrative data
Eurostat
3. Output quality assessment: input and process
Not feasible:
• multiple sources
• multiple uses
• large and complex processes
• certainly at the European level
6/11Quality indicators for statistics based on multiple sources
Mihaela Agafiţei, Fabrice Gras, Wim Kloek, Sorina Vâju(Eurostat, European Commission)
Q2014 – Vienna – 5th of June, 2014Session No 32 - Statistics beyond survey and administrative data
Eurostat
3. Output quality assessment: input and processProcess step Risk
Impacted quality dimension
Error measurement
Linkage and determination of the target population
Missed link, wrong link: under/over coverage
Accuracy, comparability
Bias, confidence range of the target population
Concept/definition
Aggregation of different concept/definitions
Relevance, accuracy, comparability
Bias, Variance error, qualitative assessment
Imputation/estimation
Estimation error Accuracy Bias, variance error
Classification Wrong classification
Relevance, accuracy, comparability below a certain level of aggregation
Bias, variance error
7/11Quality indicators for statistics based on multiple sources
Mihaela Agafiţei, Fabrice Gras, Wim Kloek, Sorina Vâju(Eurostat, European Commission)
Q2014 – Vienna – 5th of June, 2014Session No 32 - Statistics beyond survey and administrative data
Eurostat
4. Direct output quality assessment
• Direct assessment of output quality from the output itself
• Assessment of output quality with a common reference data source
• Bootstrapping
Not replacing the input + process approach
8/11Quality indicators for statistics based on multiple sources
Mihaela Agafiţei, Fabrice Gras, Wim Kloek, Sorina Vâju(Eurostat, European Commission)
Q2014 – Vienna – 5th of June, 2014Session No 32 - Statistics beyond survey and administrative data
Eurostat
4. Direct output quality assessment Direct assessment of output quality from the output itself
• time series or cross-sectional data
• breaks in series are a direct indication of bias
• revisions
• outliers
Assessment of output quality with a common reference data source
• quality survey
• additional statistics or administrative sources with considerable
conceptual harmonisation
9/11Quality indicators for statistics based on multiple sources
Mihaela Agafiţei, Fabrice Gras, Wim Kloek, Sorina Vâju(Eurostat, European Commission)
Q2014 – Vienna – 5th of June, 2014Session No 32 - Statistics beyond survey and administrative data
Eurostat
4. Direct output quality assessment
Methods derived from bootstrapping
Possible use Application Remarks Main practical problem
As primary and/or complementary data
Yes
Existence of overlapping survey data is welcome and can significantly increase the feasibility and relevance of the method
Inference on the distribution and/or generating process of the administrative data. Detection of break and outliers in time series.
Support sampling surveys
PartiallyUncertainty can be inserted by estimating false positive and negative probability
How to simulate the addition of a previously non selected unit in the replication of the sample
Auxiliary information
Yes
Modelling on how randomness is channelled through the production process
Simulation of the error caused by the imputation/estimation methods
10/11Quality indicators for statistics based on multiple sources
Mihaela Agafiţei, Fabrice Gras, Wim Kloek, Sorina Vâju(Eurostat, European Commission)
Q2014 – Vienna – 5th of June, 2014Session No 32 - Statistics beyond survey and administrative data
Eurostat
5. Conclusions
Output quality assessment through input and process quality gets too
complex in processes combining several sources, especially at the European
level
Alternative solutions should be found:
direct output assessment
a common reference source
bootstrapping
Output quality assessment:
internal use: to monitor and improve statistical production process
external use: a coherent summary of information on quality output
Assessing quality is not for free
11/11
Quality indicators for statistics based on multiple sourcesMihaela Agafiţei, Fabrice Gras, Wim Kloek, Sorina Vâju
(Eurostat, European Commission)
Q2014 – Vienna – 5th of June, 2014Session No 32 - Statistics beyond survey and administrative data