PIAAC: OBJECTIVES AND METHODSNew England Association of Colleges Conference , 2014William Thorn, Senior Analyst, OECDwilliam.thorn@oecd.org

• Give an overview of the objectives and methods of PIAAC – Why the survey was undertaken– Content– Methodology – Reporting

Purpose

Origins

• 1980s - early 1990s– interest in literacy levels of workforce– ‘competence’ movement – Developments in large-scale testing

• Development of international adult literacy/skills surveys: – IALS (1994, 1996, 1998): 21 countries

(including Ireland in 1994)– ALL (2003, 2006): 13 countries

Origins

• Work on PIAAC began in early 2000s– Updating measures to increase

relevance to the digital world– Expansion of the range of skills about

which information collected (e.g. ‘generic’ skills)

– Interest in the ‘demand’ for skills in addition to supply

– Measurement of ‘human capital’ rather than ‘literacy’

Objectives

• Design of PIAAC finalised in 2007• Broad objectives :

– Provide high quality comparable information on the level and distribution of key information processing skills in the adult population

– Show the relationship of these skills to individual and social ‘outcomes’

– Better understand the processes through which skills are gained, maintained and lost over the lifecycle

Design features: content

• Direct assessment of key information processing skills– Literacy (including reading components), numeracy, problem

solving in technology-rich environments (PS-TRE)– Linked to IALS and ALL in domains of literacy and numeracy

• Information on the use of literacy, numeracy and problem solving at work and elsewhere

• Information on use of a range of other generic skills at work– Interaction, organisation (self and others), learning and

physical skills

• Information on antecedents and outcomes of the assessed skills

Design features

• Target population – 16-65 year olds resident in national territory

• Sample: probability sample representative of target population

• Household survey• Computer delivery

– BQ – CAPI– Assessment - CBA

Participation

• Round 1 (2008-2013)– 24 countries– Australia, Austria, Belgium (Flanders), Canada,

Czech Republic, Denmark, Estonia, Finland, France, Germany, Ireland, Italy, Japan, Korea, Netherlands, Norway, Poland, Slovak Republic, Spain, Sweden, UK (England, Nth Ireland), US, Cyprus, Russian Federation

• Round 2 (2012-2015)– 10 countries– Chile, Greece, Indonesia, Israel, Lithuania, New

Zealand, Singapore, Slovenia, Turkey

Innovative elements

• Proficiency in information processing in ICT environments – Reading of digital texts– Problem solving in technology rich environments

• Information regarding poor readers– Reading components

• Richer information regarding the use of skills– In particular, the use of generic skills

• Computer delivery

Literacy

The ability to...Understand, evaluate, use and engage with written texts.

In order to..Achieve one’s goals, and to develop one’s knowledge and potential.Literacy encompasses a range of skills from..The decoding of written words and sentences The comprehension, interpretation and evaluation of complex texts.

Numeracy

The ability to…Access, use, interpret and communicate mathematical information and ideas

In order to.. Engage in and manage the mathematical demands of a range of situations in adults. Numeracy involves Managing a situation or solving a problem in a real context, by responding to mathematical content/information/ideas represented in multiple ways.

Problem SolvingIn Technology-rich Environments

The ability to…Use digital technology communication tools and networks to acquire and evaluate information, communicate with others and perform practical tasks.

The assessment focuses on the abilities to…Solve problems for personal, work and civic purposes by setting up appropriate goals and plans, and accessing and making use of information through computers and computer networks.

10

The skills assessed

How the domains are defined

• Content – the artefacts, tools, knowledge, representations and

cognitive challenges that constitute the corpus adults must respond to or use

• Cognitive strategies – the processes that adults must bring into play to

respond to or use given content in an appropriate manner

• Contexts – the different situations in which adults have to read,

display numerate behaviour, and solve problems

Section Content

A Basic demographics (age, sex)B Educational attainment and participationC Labour Force status and work historyD Characteristics of current jobE Characteristics of last job (if unemployed and worked in

last 5 years)F Skill use at work (generic skills)G Skill use at work (literacy, numeracy and ICTs)H Skill use in everyday life (literacy, numeracy and ICTs)I ‘Social outcomes’ and learning strategiesJ Background characteristics (social, linguistic and

migration background and family composition)

The background questionnaire

Reporting

• Results presented on a 500 point scale – test items and test-takers located on the

same scale– Difficulty of items– Proficiency of persons

Figure I.2.5Relationship between difficulty of assessment items and proficiency of adults on literacy,

numeracy, problem solving in technology-rich environments scalesFigure I.2.4 Foreign-born population as a % of total population

Proficiency levels

• To help interpret results the scale divided into proficiency levels

• Descriptors developed to summarise the underlying characteristics of items in each level in terms of the literacy framework

• Descriptive not normative • 6 proficiency levels in literacy and

numeracy and 4 levels in PSTRE

Proficiency levels: literacy and numeracy

Level Score Range

Less than level 1 Lower than 176

Level 1 176-225

Level 2 226-275

Level 3 276-325

Level 4 326-375

Level 5 376 and above

Proficiency levels: numeracy

Level Task features

Less than level 1 Use simple processes (e.g. counting, basic arithmetic, recognising spatial representations; familiar contexts; little text and few distractors

Level 1 Use basic mathematical processes (e.g. involving one step): counting sorting, simple percentages, identifying elements of common graphical or spatial representations

Level 2 Use mathematical process involving two or more steps (calculations with whole numbers and decimals, percentages and fractions, simple measurement); common contexts; content explicit and few distractors

Level 3 Use mathematical processes involving several steps and choice of problem solving strategies; application of number and spatial sense; patterns and proportions, statistical data in texts tables and graphs

Level 4 Understand complex and abstract mathematical information; tasks require analysis and more complex reasoning

Level 5 Understand complex representations and mathematical and statistical ideas; integrate multiple types of mathematical information; draw inferences, work with models and evaluate choices and solutions

Level 4: numeracy

Tasks at this level require the respondent to understand a broad range of mathematical information that may be complex, abstract or embedded in unfamiliar contexts. These tasks involve undertaking multiple steps and choosing relevant problem-solving strategies and processes. Tasks tend to require analysis and more complex reasoning about quantities and data; statistics and chance; spatial relationships; and change, proportions and formulas. Tasks at this level may also require understanding arguments or communicating well-reasoned explanations for answers or choices.

Proficiency levels: literacy

Level Task features

Less than level 1 Read short texts on familiar topics; few distractors; information requested is identical with that in text; basic vocabulary knowledge; no knowledge of text features required

Level 1 Read relatively short texts to locate information that is identical to or synonymous with information requested; few distractors; cycling though more than one piece of information; basic vocabulary; some knowledeg of text features

Level 2 Make matches between text and information, paraphrasing, low level inferences; some distractors; use strategies such a cycling through information, compare and contrast, navigate through digital texts

Level 3 Read relatively long and dense texts; understand text and rhetorical structures; evaluate information and make inferences; multi-step operations; distracting information present

Level 4 Read complex and lengthy texts; complex inferences and background knowledge required; conditional and competing information present

Level 5 Search for and integrate information across multiple and dense texts; construct syntheses of similar and contrasting ideas and points of view; evaluate evidentiary sources; awareness of rhetorical cues; make high level inferences and draw on specialise background knowledge.

Level1: Literacy

Most of the tasks at this level require the respondent to read relatively short digital or print continuous, non-continuous, or mixed texts to locate a single piece of information that is identical to or synonymous with the information given in the question or directive. Some tasks, such as those involving non-continuous texts, may require the respondent to enter personal information onto a document. Little, if any, competing information is present. Some tasks may require simple cycling through more than one piece of information. Knowledge and skill in recognising basic vocabulary determining the meaning of sentences, and reading paragraphs of text is expected.

Find Out More about PIAAC at:

www.oecd.org/site/piaac All national and international publications

The complete micro-level database

Emailwilliam.thorn@OECD.org

Thank you