PIAAC: Delivering the Survey in Ireland

Kevin McCormackSenior Statistician - Social Analysis

www.PIAAC.iePIAAC@cso.ie

Overview

• Background• Development• Survey Process• Content• Results• Dissemination

HandoutsPIAAC BrochureBackground QuestionnairePIAAC FactsheetOutline of international reportProposed thematic reports

• SLA 2010 - 2013– DES (and DETE originally) – CSO

• Statistics Act 1993– Data collection– Dissemination

• Timeline– 2009 – 2013 – Publication Oct 2nd 2013

Background

June 2010 – July 2011Main Study preparation:

TestingDeveloping IT systems Commissioning laptops

Training interviewers

PIAAC Timeline

2009 2010 2011 2012 2013

September 2009 CSO commenced formal engagement

Translation and Localisation tasks began

March 2010SLA signedCSO, DES, DETE

May 2010Field TrialInterviewer trainingSampling, maps, addressesQuestionnaire testing

October 2013PublicationInternationalNational&Databases

2009 2010 2013

May 2012 – May2013Data processing,

verification, cleaningTable generationNational report

August 2011 – March 2012Main Study

5,983 completed interviews72% Overall response rate70% Computer 30% Paper

• Instrument development• Translation and Localisation• Scoring testing and re-testing

• IT development• Virtual Machine • Screener and Case Management System development

• Field Test

Survey Development

Survey Development

• OECD oversight– “The Consortium”

• Sampling plans• Interviewer training• Quality Control• Weighting - Westat• Data Adjudication

• Main Study Aug 2011 – March 2012

• Sampling• Interviewer training• Data collection• Data processing

Survey Process

700 Blocks 15 Households 1 Person

Randomly

selecte

d !

!!

• Main Study• Sampling• Interviewer training• Data collection• Data processing

Survey Process

• Mix of new and experienced interviewers• 8 day training programme

• OECD specification

Focus on :•Gaining co-operation•Use of survey instrument

•Other important elements:•Nature of survey• Quality control•Confidentiality

• Main Study• Sampling• Interviewer training• Data collection• Data processing

Survey Process

•Encrypted data - file transfer & laptops

•Screener - random adult (16-65)•Targets and bonus payments•Incentive payment

•Management information system•Unlimited call –backs•Strict weekly monitoring

Quality control•Taped interviews•10% of all interviews verified•Telephone calls, field visits,

• Main Study• Sampling• Interviewer training• Data collection• Data processing

Survey Process

Industry and occupation codingManual AutomaticVerification against LFS

Paper booklet scoring Inter-coder reliability designAnchor booklets (international)

Data cleaningScreener vs PIAAC questionnaire

Survey Content

• Background Questionnaire

• ‘Core’– IT competency– Basic Skills assessment

• Direct Assessment– Computer Based Assessment– Paper Based Assessment

Background QuestionnaireA. General Information – Age & Gender

B. Education and Training – Formal Education & Training

C. Current status and work history

D. Current work – Occupation & Earnings

E. Last job - Occupation & Earnings

F. Skills used at work – Frequency of use of workplace skills

G. Literacy, numeracy and ICT skills used at work

H. Literacy, numeracy and ICT skills used in everyday life

I. About yourself – Respondent attitudes and activities

J. Background information – including parents’ occupations

FAIL <4 of 8

FAIL <3 of 6

PIAAC Assessment Design

BQ

Basic Skills (Computer)

Basic IT Skills

Assessment (Computer)

Reading Components

Basic Skills (Paper)

Assessment (Paper)

Everyone takes the BQ

Aged 16-65

6 tasks: 4 of 6 to PASSScrolling, clicking, highlighting,

4 literacy tasks4 numeracy tasks(same as computer tasks) 4 of 8 to PASS

FAIL

<4

3 literacy tasks3 numeracy tasks

3 of 6 to PASS

•Literacy and Numeracy•Numeracy and Literacy •Numeracy and Problem-solving

Literacy OR Numeracy booklet

Selected at random

•All those who fail the simple literacy or numeracy task on computer or paper are routed automatically to the Reading Components booklet

PIAAC Assessment Design

BQ

Basic Skills (Computer)

Basic IT Skills

Assessment (Computer)

Reading Components

Basic Skills (Paper)

Assessment (Paper)

Respondents had the option of skipping the Core IT assessment and proceeding directly to the Paper route

Results

Background Questionnaire• Economic• Education• Social

• Age• Gender

• Skills use (work & home)

Assessments• Literacy• Numeracy• Problem-solving in

Technology Rich Environments (PS-TRE)

• Proficiency – Item difficulty– Item Response Theory & scale scores– Cut-points (levels 1 to 5)

• IRELAND: Trends from IALS to PIAAC (linked items)

PIAAC Proficiency Scales

Trends (IALS vs PIAAC)

IALS ALL PIAAC

Prose Literacy Prose Literacy

Document Literacy Document Literacy

Literacy (rescaled for trend) Literacy (rescaled for trend) Literacy

Quantitative Literacy

Numeracy Numeracy

Problem solving

Problem solving in TRE

PIAAC Proficiency Scales

Level 1Level 2Level 3Level 4Level 5

0-225

226-275

276-325

326-375

376-500

Key findings of IALS

• 25% of Irish Adults at lowest levels of ‘literacy’

• Gender: Males more likely to be at higher levels in Quantitative Literacy – otherwise little difference between males and females

• Age: Older people more likely to be at lower levels of ability – related in IALS to educational attainment

• Education: Higher performance related to higher levels of educational achievement.

Prose Literacy (IALS 1997)Document Literacy (IALS 1997)

Level 1 Level 2 Level 3 Levels 4/50%

5%

10%

15%

20%

25%

30%

35%

40%

23%

30%

34%

14%

25%

32% 32%

12%

25%28%

31%

16%

Scales

Quantitative Literacy (IALS 1997)

IALS 1997

Canad

a

German

y

Irelan

d

Netherl

ands

Poland

Swed

en

Switz

erlan

d (Fren

ch sp

eakin

g)

Switz

erlan

d (Germ

an sp

eakin

g)

United St

ates

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

18.29

25.310.1

45.4

6.216.2 18.1 23.7

25.1 18.911.5

205.8

35.5

16 16.1 19

IALS 1997

International comparison Documents scale

International comparison Quantitative scale IALS 1997

Some data!

1996 2011

77.3

87.6

72.0

85.382.8

89.8

Percentage of those 20-24 with Leaving Certificate or equivalent

Both sexes Male Female

Source: CSO (Census 1996 & QNHS Q2 2011)

Source: DES (Education Statistics 2010/2011)

Highest level of Education Attained (Aged 25-64)

Primary (incl. no formal education)

Lower secondary Upper secondary Third level: Non-degree Third level: Degree or higher

25%

22%

31%

10%12%

10%

15%

24%

28%

23%

1996 2011

Source: CSO (Census 1996 & QNHS Q2 2011)

Population change 1996 to 2011

0 - 4 years

5 - 9 years

10 - 14 years

15 - 19 years

20 - 24 years

25 - 29 years

30 - 34 years

35 - 39 years

40 - 44 years

45 - 49 years

50 - 54 years

55 - 59 years

60 - 64 years

65 - 69 years

70 - 74 years

75 - 79 years

80 - 84 years

85 years and over

0

50000

100000

150000

200000

250000

300000

350000

400000

450000

Age profile of Population 1996 vs 2011

2011 1996

+1m (26%)

Country of Birth

Source: CSO (Census 1996 & 2011)

Ireland; 93%

Outside Ireland; 7%

1996

Ireland; 80%

Outside Ireland; 20%

2011

PIAAC results

International report • 2nd October 2013• Two volumes

– Results (I) and Technical (II)

• Distribution of skills (mean scores)

• International comparisons• Main focus on LITERACY• Other domains on the web• Problem-Solving in TRE for

later thematic releases

National report• Focus on Irish

performance

• Trends from IALS

• Relevant international comparisons

Profile 9 – Census 2011

A Study of Education & Skills in Ireland• November 22nd 2012

Thank you

Questions?