Continuous improvement of EU-SILC quality: standard error estimation and

new quality reporting system

Emilio Di Meglio and Emanuela Di Falco (EUROSTAT)

Q2014 Conference Vienna 1

Why variance estimation? Requested by regulation

Quality report Compliance

Requested by users Policy relevance of indicators

Requested by researchers

Q2014 Conference Vienna 2

Current legal requirements According to Reg.1982/2003, the X and L (initial sample) data

are to be based on a nationally representative probability sample of the population residing in private households.

Representative probability samples shall be achieved both for households and for individual persons in the target population.

The sampling frame and methods of sample selection should ensure that every individual and household in the target population is assigned a known and non-zero probability of selection.

Reg. 1177/2003 defines the minimum effective sample sizes to be achieved.

Q2014 Conference Vienna 3

Main challenges for EU SILC Difficulty to find the « best » possible method for

variance estimation at Eurostat level– Different designs (flexibility)– Missing information– Debate on methods ongoing

Differentiate the needs: accuracy estimates for policy usage and accuracy estimates for researchers.

Q2014 Conference Vienna 4

Sampling design by country (2012)

Q2014 Conference Vienna 5

Sampling design Country Without stratification

Simple random sampling MT ,DK, IS Systematic sampling SE,NO

With stratification Stratified sampling according to different design by rotational group HU

Stratified simple random sampling LU, CY, SK, CH, LT, DE*,AT Stratified and systematic sampling EE Stratified multi-stage sampling CZ, ES, PL,RO,IE Stratified two-stage clustered sampling PT Stratified two-stage systematic sampling SI, NL, HR Stratified multi-stage systematic sampling FR, LV, UK, BE, BG, EL, IT Stratified two-phase sampling FI * from former participants of micro census

Our objective Resampling taking into account all the possible elements

coming from 32 countries would be extremely computationally and resource intensive

Variance estimation methods balancing between scientific accuracy and administrative considerations (time, cost, simplicity) are the only viable solution

Aim: to quickly provide to users and policy makers standard errors for the SILC-based indicators, particularly the AROPE (At-Risk-Of-Poverty or social Exclusion), its components and its main breakdowns.

Q2014 Conference Vienna 6

The method (synthesis) Linearization is a technique based on the use of linear approximation to

reduce non-linear statistics to a linear form, justified by asymptotic properties of the estimator (Särndal et al, 1992 ; Deville, 1999 ; Wolter, 2006 ; Osier, 2009)

The "ultimate cluster" approach (Särndal et al, 1992) is a simplification consisting in calculating the variance taking into account only variation among Primary Sampling Unit (PSU) totals

This method requires first stage sampling fractions to be small which is nearly always the case.

This method allows a great flexibility and simplifies the calculations of variances.

It can also be generalized to calculate variance of the differences of one year to another (Berger, 2004 , 2010 ).

Applicable with the main statistical packages (SAS, R, STATA)

Q2014 Conference Vienna 7

Results on AROPE For 6 countries 95% Confidence Interval for AROPE equal or

smaller that ±1.0% (CZ, IT, SI, DE, FI, NO)

For 9 countries 95% Confidence Interval for AROPE between ± 1% and ±1.5% (ES, PL, UK, EE, AT, SK, CH, SE, IS)

For 8 countries 95% Confidence Interval for AROPE between ±1.5% and ±2% (BE, DK, HR, HU, NL, PT, CY, MT)

For 6 countries 95% Confidence Interval for AROPE larger than ±2% (BG, EL, IE, RO, LT, LV)

Complete results in EU-SILC quality reportQ2014 Conference Vienna 8

Measurement of net changesTo measure the significance of the evolution of social indicators

Example: When the At-risk-of-poverty or social exclusion rate for Poland goes from 27.2% in 2011 to 26.7% in 2012, are we able to say that this change is significant?

Exercise already done for AROPE and other main EU-SILC indicators

Q2014 Conference Vienna 9

Output

Q2014 Conference Vienna 10

CountryAROPE (2011)

%

AROPE (2012)

%

Difference 2012 – 2011

(% points)

Standard error

(% points)

Significance of change

HU 31.0 32.4 1.5 0.7 YMT 21.4 22.2 0.8 0.4 YNL 15.7 15.0 -0.8 0.2 YPL 27.2 26.7 -0.5 0.3 N

EU-SILC Quality reports(Reg. No 1777/2003)

At national level, Member States have to produce: • An Intermediate QR (by the end of the year N+1)Based on cross-sectional data of year N• A Final QR (by the end of the year N+2)Based on longitudinal and cross-sectional data year N

At European level, EUROSTAT has to produce :• EU Comparative Intermediate QR (by June of the year N+2)Based on the national Intermediate QRs• EU Comparative Final QR (by June of the year N+3)Based on the national Final QRs

Q2014 Conference Vienna 11

Quality reporting Revision process• New template (ESQRS)

• Revision of the Contents• Introduction of annexes and questionnaire

• ESS Metadata Handler (old NRME)

Q2014 Conference Vienna 12

EU SILC key quality dimensions• Accuracy• Comparability• Coherence National ESQRS• Cost and burden• Statistical processing

• Timeliness and punctuality• Relevance EU ESQRS

Q2014 Conference Vienna 13

Availability of quality metadata• Quality reports• Questionnaires• Methodological papers

Further action: integrate more information in a wiki platform

Q2014 Conference Vienna 14

Conclusion and future plans The variance estimation methodology is of relatively simple

application It can be considered as a good compromise between scientific

soundness and feasibility under current constraints. The next steps consist in still improving these calculations by asking

Member States to provide the necessary information where missing. Dissemination of further information to users. Better disseminate quality reports

Q2014 Conference Vienna 15