Measuring the quality of regional estimates from the ABSJennie Davies and Daniel Ayoubkhani

Overview

• ABS• User needs• ABS estimation• Methods• Results• Recommendations• Outcomes

Annual Business Survey (ABS)

• ONS’s largest business survey• Sample of ~73,000 UK businesses• Covers agriculture (part), production,

construction, distribution and service (part) sectors

• Variables (collected and derived) include:TurnoverPurchases of goods and servicesApproximate Gross Value AddedNet capital expenditure

Annual Business Survey (ABS)

• National publication (November and June)4-digit SIC breakdown

• Regional publication (July)12 UK regions, 2-digit SIC breakdown

• Special Analysis systemAd-hoc requests for lower level estimates (eg local

authority, 3-digit SIC by employment sizeband etc)

• Standard errors and CVs provided with national estimates but not regional

User needs

• Large range of ABS usersCentral government, local government and devolved

administrations, Eurostat, National Accounts, academia, consultancy firms, media, public

• Quality measures therefore important for sound decision making

• Lack of standard errors for regional estimates makes it difficult for users to assess accuracy

User needs

• Recent UKSA Assessment of ABS:“there is insufficient information about methods and

quality”

“there is no information about the resulting quality of the statistics and no caveats around their use”

• QIF project therefore undertaken to develop methodology for calculating standard errors of published regional ABS estimates

• Aim to provide users with information about the quality of regional estimates

Reporting unit data

National estimates

Modelled local unit data

Regional estimates

ABS estimation

Regional apportionment model

Ratio estimation

Ratio estimation

National standard errors

GES

Regional standard errors

???

Small area estimation for small domains

7

Standard error estimation (1)

• Assume that the apportioned values are “true” returns

• Single stage cluster sampling• Use GES to calculate standard errors

Standard error estimation (2)

• The regional apportionment model parameters depend on the sample data so are variable

• Use bootstrapping to capture the use of the regional apportionment model

Bootstrapping

• Standard errors capture sampling variability ie how much estimates vary under different possible

samples

• Bootstrapping re-samples from the original sample to create a new sample

• Carry out estimation on new sample• Repeat lots of times• Calculate standard error of the resulting

estimates

Bootstrapping

• Fix the model parameters based on the full sample dataquality assure method against GES

• Re-fit the regional apportionment model in each iterationinclude (possible) additional variance from the model

Results

• Compared the methods in terms of:Differences in standard errors

Practical considerations

• Results for turnover presentedOther variables produced similar results

GES vs bootstrap without re-fitting the model

• Bootstrapping without re-fitting the regional apportionment model in each iteration should be comparable with estimates from GES

Bootstrapping with and without re-fitting the model

• Comparing these to see if the regional apportionment model increases variances

• Compared to bootstrap rather than GES to remove additional differences seen before

Practical implications

• Bootstrapping took ~30 hours• Relied on exact replication of the regional

apportionment model Problem for derived variables such as aGVA

Recommendations

• Use GES to produce standard errors• With caveats that the regional apportionment

model is assumed to be fixed

Outcomes

• Report published on ONS website (Feb 2014)• Methodology approved by ABS Survey

Management Board• Standard errors of regional estimates to be

published for first time in July 2014• Another QIF bid submitted to investigate

standard errors of small area ABS estimatesWould allow for quality measures to be published

alongside majority of Special Analysis requests

Summary

• ABS• User needs• ABS estimation• Methods• Results• Recommendations• Outcomes