Harmonisation: Understanding user experiences to maximise

the value of statistics

Rachel LeeserSenior Research and Statistical Analyst - Social Exclusion Intelligence,

Greater London Authority rachel.leeser@london.gov.uk

Emma Emery User Engagement Manager, Royal Statistical Society (UK Stats Authority

secondment) e.emery@rss.org.uk

What we’re going to talk about…

• Reminder of Code of Practice & link between harmonisation, user engagement and maximising value of stats

• Examples of current harmonisation issues

• Rachel’s experience using income & earnings statistics

• Consideration of impact of lack of harmonisation/info about harmonisation

• What the GSS should be doing & examples of GSS harmonisation work

Harmonisation in 2015 EU peer review report - recommendation

For full details see p24 of the report here: http://ec.europa.eu/eurostat/documents/64157/4372828/2015-UK-report/d44f7d3f-64c1-4450-8a37-bfadb8542607

15.The United Kingdom Statistics Authority and the National Statistician as Head of the Government Statistical Service should strengthen their efforts to harmonise United Kingdom statistics in order to ensure coherence and comparability in the production of official statistics within the United Kingdom, over time and among producers of statistics. (European statistics Code of Practice, Principles 10, 14 and 15, coordination.)

Harmonisation: Relevant aspects of the UK Code of Practice for Official Statistics (1)

‘Promote comparability within the UK and internationally by, for example, adopting common standards, concepts, sampling frames, questions, definitions, statistical units and classifications (including common geographic referencing and coding standards). Make the reasons for any deviations from standard models publicly available.’

Harmonisation: Relevant aspects of the UK Code of Practice for Official Statistics (2)‘Ensure that official statistics are disseminated in forms that

enable and encourage analysis and re-use. Release datasets and reference databases, supported by documentation, in formats that are convenient to users.’

‘Seek to balance quality (for example, accuracy and timeliness) against costs (including both costs to government and data suppliers), taking into account the expected uses of the statistics.’

• This translates into a fair bit of work for statistics producer teams – some more than others - but is important for ensuring that statistics are as useful and valuable as possible

• Ongoing communication with statistics users essential in truly adhering to these aspects of the Code of Practice

Harmonisation in assessment reports - exampleNI Housing and Benefits stats. User feedback reported:‘Concerns raised by housing reports users included uncertainty about

the definitions of some classifications, a lack of coherence between specific statistics and those from other sources and a lack of explanation behind the data.’

‘The availability of local data and the coherence of benefits statistics with those from DWP were cited as being particularly useful.’

Recommendations:

(Sugg) ‘Provide more information about the availability of comparable housing statistics for England, Scotland, Wales and the Republic of Ireland.’

(Sugg) ‘Work with DWP to provide comparisons and analysis of benefits statistics in Northern Ireland and in GB that will aid user interpretation.’

Other examples of lack of harmonisation affecting users…(1)Some Census questions differ between UK countries – e.g

religion, resulting in questions over the impact on the data - is it why Scotland measured a substantially lower proportion of Christians & a substantially higher proportion of ‘no religion’ than E&W census?:

Other examples of lack of harmonisation affecting users…(2)

• Other Census issues – different age bands used by UK countries & timeliness varies. Analysis by age bands for whole of UK difficult.

• NHS health data e.g. waiting times not comparable between UK countries

• Differences between SOC2000 & SOC2010 causing break in time-series – can mean longitudinal series impossible e.g. Certificate of Sponsorship data

To harmonise or not to harmonise Income, Earnings, Deprivation & Poverty statistics - a user’s perspective

• Deprivation, Earnings, Income, Poverty– Nebulous concepts– Constantly evolving– Variation between areas/commmunities

• Accuracy

• What would harmonisation look like?

Earnings – what counts?

• “Basic”, “Total”, “Net”, “Gross”, “Average”– What to include?– Overtime– Bonus/honorarium– Tips– In kind payments/benefits – accommodation, meals, pension contributions,

discounts, perks– Deductions – travel, uniforms– Variable contract or zero hours– Self-employed

• Hourly, Weekly, Annual etc• Sick pay, leave, bank holidays• Main job, additional jobs

Earnings – Sources

• ASHE Annual Survey of Hours and Earnings– Employer survey of jobs, not people

• LFS – Labour Force Survey– (APS) self-reported hours and earnings

• FRS – Family Resources Survey– (HBAI) self reported weekly

• EHS – English Housing Survey• WAS – Wealth and Assets Survey• STES – Short Term Earnings Survey• Monthly Wages and Salaries Survey

– (AWE) employer reports, best for bonuses

Earnings – why do we want to know?

• Comparative analysis, eg– Equalities– Regional– Occupation, industry

• Proxy• Spending potential

– with other (inferred/imputed?) income

• Living Wage

Earnings – deriving meaning

• Co-variables– geography– full-time/part-time, self-employed– ………….

• Household size/characteristics• ??costs directly associated with

working– travel– childcare

Deprivation

• No direct measure of something we can call “Deprivation”

• Indicators– Variation over time and space

• Indices– CLG/Wales/Scotland/NI– Census– Harmonise domains?

• Flexibility rather than harmonisation

Income

• Sources FRS, Small Area Estimates, WAS, GDHI, Effects of taxes and Benefits, LCF, Commercial datasets……………..

• Gross or net– Net of what?– Housing Costs

• Banded – equal or variable – Ability to relate to household members (equivalisation)– Ability to relate to expenditure

• Particularly housing affordability

• Fuel costs etc

• Transfer payments

• Reporting – HBAI reports for people in households, ONS small area estimates report for households

Poverty

• Wealth or income?• Relative or “absolute”

– 60% of median of what?

• Equivalisation scales• Disability (this applies elsewhere too!)

• Distinction between poverty and deprivation?– Child poverty measures in particular

To harmonise or not to harmonise- some questions to ponder

• What are the merits of:– flexibility vs consistency?– transparency and detail vs easy?– change vs continuity

• Local, national or international comparability?

Do they have to be either/or?

So… potential impact of lack of harmonisation/info about harmonisation:

• Confusion for users of statistics

• Difficult or impossible to answer research questions

• Time consuming for users to identify the differences & consider the implication for use

• Perception that official statistics are complicated or confusing can lead to lack of trust

• Statistics/datasets don’t get used as much as they might be if greater harmonisation – therefore, not as valuable, or having as much impact as they could be

• BUT in some cases steps to increase harmonisation may not be welcomed by users!

Examples of GSS promoting harmonisation & responding to users• ONS responded to user feedback on the 2001 Census and put in

place for 2011 outputs: Common table specifications for the key & quick statistics tables (users

previously had to do the table matching themselves) Standard bulk delivery file format for UK (users previously had to create a

common format themselves prior to analysis work)

• Personal well-being – ONS developed 4 standard questions & added them to the Harmonisation Programme Standards to encourage use by other research organisations. But EU-SILC uses different questions, meaning comparisons with other countries not straightforward.

• Disability – harmonised survey questions created following GSS project involving stakeholders. More info: http://www.ons.gov.uk/ons/guide-method/harmonisation/primary-set-of-harmonised-concepts-and-questions/index.html Will they/are they being used?

Actions for you….• Ensure external communication and researching the

uses & impact of statistics are key elements of the production process

• Encourage increased UK/international harmonisation through collaboration with other statistics producer teams

• Consider the impact of methodological changes on longitudinal series that are valuable for research

• Make use of community sites such as StatsUserNet to investigate users’ experiences of statistics & inform balancing of priorities

• Ensure info available about extent of harmonisation & comparability, and deviation from common standards

Useful references….

• GSS Guidance on assessing & improving comparability: https://gss.civilservice.gov.uk/statistics/presentation-and-dissemination/comparing-official-statistics-across-uk/

• GSS Primary set of harmonised concepts and questions: http://www.ons.gov.uk/ons/guide-method/harmonisation/primary-set-of-harmonised-concepts-and-questions/index.html

• National Statistics Harmonisation Group: https://gss.civilservice.gov.uk/about/governance-and-structure/national-statistics-harmonisation-steering-group/

• Quality, Methods & Harmonisation Tool: https://gss.civilservice.gov.uk/statistics/methodology-quality/quality-2/quality-resources/quality-methods-and-harmonisation-tool/

• GPT spreadsheet guidance: https://gss.civilservice.gov.uk/blog/2014/11/new-gss-spreadsheet-guidance-released/

Get in touch!• ONS harmonisation team for advice on improving

harmonisation of statistical products: harmonisation@ons.gsi.gov.uk

• Emma Emery or the GSS Good Practice Team about improving external communication & engagement: e.emery@rss.org.uk / goodpracticeteam@statistics.gsi.gov.uk

• Rachel Leeser on user experiences: Rachel.Leeser@london.gov.uk

• And remember about StatsUserNet! www.statsusernet.org.uk @StatsUserNet