PIAAC=

Programme for the International Assessment

of Adult Competencies

• Through the assessments and background questionnaire, PIAAC situated respondents within a continuum of life-long learning

1994International Adult Literacy

Survey (IALS)

2003 Adult Literacy and Life Skills

Survey (ALL) – in Canada IALSS

2011 Programme for the

International Assessment of Adult Competencies

(PIAAC)

4.

From Whence We Came

PISA• The OECD also coordinates the Programme

for International Student Assessment (PISA) Every 3 years, starting 2000 15 year old students from randomly selected schools

worldwide Assessments in reading, mathematics and science Students and principals complete background

questionnaires looking at family background, approaches to learning and school administration

In some cases, parents also complete a questionnaire

What PIAAC Assessed

Literacy• Single scale measuring reading of prose,

document, digital and mixed format texts• IALSS reported literacy as two separate domains (prose and

document comprehension on two separate scales)

What PIAAC Assessed• Numeracy

Essentially the same concept of numeracy as used in IALSS

PIAAC, however, gathered significantly more data for the construction of its numeracy scale

What PIAAC Assessed• Problem Solving in a Technology Rich

Environment (PS-TRE)

PS-TRE• Newly introduced domain

• PIAAC broadens the concept of literacy by including digital technology, communication tools and networks to acquire and evaluate information, communicate with others, and perform practical tasks

• At a basic level, it represents the intersection between computer literacy skills and the cognitive skills required to solve problems

• PIAAC assessed adults capacity to use computer tools (keyboards and mice, file management tools, applications, and graphic interfaces) to assess the capacity of adults to access, process, evaluate and analyze information effectively

• PIAAC employed adaptive testing – a feature of computer-based testing that adjusts the difficulty of questions as participants take the test

• Provides a much more nuanced distinction in scores than were provided by paper-based predecessors

• If respondents could not (or would not) use the computer based instrument, they were provided with a paper-based instrument that measured literacy and numeracy

• Respondents could score at levels 4/5 in literacy and numeracy using the paper-based instrument

Direct assessment (skills measured by the survey)• Literacy• Numeracy• Problem solving in technology-rich environments

Background questionnaire (information about respondents)• Demographic characteristics • Educational attainment and training• Employment status and income • Social and linguistic background

Module on skills use (information on how skills are used every day)• cognitive skills (e.g., engagement in reading)• non-cognitive skills (e.g., capacity to work in a team) and communications, organization, and

influencing skills• workplace skills (e.g., autonomy over key aspects of work)

Main Elements of PIAAC

The Background• 157,000 adults 16-65 in 24 countries

Second round with nine additional countries Third round proposed for 2014

• Language of assessment was the official language(s) of the countries participating

• Sample sizes were determined nationally• Canada chose a larger sample size to

ensure PIAAC data could be compared to IALSS

PIAAC Sample By Country

PIAAC Sample in Canada General

sampleAboriginal

populationsMinority- language

populationsRecent

immigrants Youth Total

British Columbia 974 302 649 508 2,733Alberta 947 100 177 1,224Saskatchewan 994 607 1,601Manitoba 990 624 698 2,312Ontario 2,758 1,168 647 740 5,313Quebec 5,311 60 314 226 5,911New Brunswick 1,230 456 1,686Nova Scotia 1,441 1,441Prince Edward Island 929 929Newfoundland and Labrador 1,609 1,609Provinces only 17,183 3,161 2,115 1,792 508 24,759 Nunavut 185 594 779Northwest Territories 463 454 917Yukon 471 359 830Territories only 1,119 1,407 2,526 TOTAL 18,302 4,568 2,115 1,792 508 27,285

15

PIAAC Sample in Canada• Aboriginal status was self-declared, as determined

through other Stats Canada reporting• Hours and hours of pan-Canadian conference

calls and meetings were required to achieve this level of partipation• Conference calls and meetings continue as we start to

work with the data• Confidentiality assured. The Ministry of Education

will have access to the full data set and this was communicated to respondents

The Results …

Computer Based Assessment

Computer Based Assessment• At 86%, Saskatchewan had the third

highest proportion of the population (in the world) that engaged in computer based survey – after Sweden and the Netherlands.

• The next highest proportion in Canada was Nova Scotia at 84%

Computer Based Assessment• Japan had a relatively low rate of

completion via computer, BUT scored highly in literacy and numeracy PS-TRE could not be measured by the

PIAAC paper-based assessment instrument

Saskatchewan is …• At the OECD average in literacy

• At the OECD average in PS-TRE

• Below the OECD average in numeracy

Saskatchewan …• Saskatchewan’s overall performance

mirrors Canada’s performance• Matches the OECD average for people at

the highest proficiency levels• Has a higher proportion of the population

at the lowest proficiency levels in numeracy and PS-TRE compared to the OECD average

Saskatchewan … • Had 10% of respondents whose first

language was not English and French Matches the OECD average, but is 13%

below the Canadian average• Youth (16-24) is performing below the

OECD average in all three domains In PS-TRE, Saskatchewan youth at levels

2/3 is 38% vs. the OECD average of 51%

• Nationally, educational achievement for both Aboriginal and non-Aboriginal people affects their overall literacy attainment in similar ways Saskatchewan’s results do not show the same correlation, and

we are starting to work with Manitoba and other pan-Canadian partners to try and understand why the data shows this

• The gap between Aboriginal and non-Aboriginal populations in Saskatchewan (at a 26 and 35 score point difference in literacy and numeracy respectively), is larger than in Canada overall at 14 and 22 points.

Proficiency Levels - Literacy

Below Level 1• 3% OECD, 4% Canada, 3% Saskatchewan

Level 1• 12%, 13%, 14%

Level 2• 34%, 32%, 33%

Level 3• 39%, 38%, 39%

Level 4/5• 12%, 14%, 11%

Proficiency Levels - Numeracy Below Level 1

• 5% OECD, 6% Canada, 6% Saskatchewan Level 1

• 14%, 17%, 18% Level 2

• 33%, 32%, 33% Level 3

• 35%, 33%, 33% Level 4/5

• 13%, 12%, 11%

Proficiency Levels - PS-TRE Non-respondents

• 24% OECD, 19% Canada, 15% Saskatchewan Below Level 1

• 12%, 15%, 18% Level 1

• 29%, 30%, 35% Level 2

• 28%, 29%, 28% Level 3

• 6%, 7%, 5%

• So Brett, how does our performance compare to IALSS? (and, wink, wink, I know we’re doing

worse)

PIAAC and IALSS• It is not possible to make a direct comparison of the results

from IALSS and PIAAC; the scales used are not the same• The data from the prose, document and numeracy

measures of IALSS have been rescaled at the national level to fit with the literacy measure used in PIAAC – these can be compared, but are still do not represent a 1 for 1 correlation

• Level 3 in PIAAC is not equivalent to Level 3 in IALSS. In IALSS the OECD identified Level 3 as a minimum threshold of the skills needed to manage the demands of work and life. IN PIAAC, Level 3 was not defined the same way.

PIAAC and IALSS• The OECD designed the PIAAC survey to help jurisdictions

determine the degree to which their skills address their needs. As a result, the OECD proposes that results for each jurisdiction should be assessed in terms of the association between skill levels and other characteristics, such as job, level of education, and income specific to the populations in each jurisdiction.

• Does that mean all of our existing benchmark work is now thrown out? No – at this stage, the results from PIAAC have limited bearing on existing benchmarks. Much further analysis and research will be needed before the results can be applied to literacy programs and the measures we all use.

• Overall, the difference between the performance of people from different backgrounds within jurisdictions is much larger than the difference between jurisdictions themselves

• e.g., The gap between Canadians (and Saskatchewanians) at the 95th and 5th percentiles is larger than the gap between OECD countries on average

Skills in the labour forceLiteracy ― Proficiency of population aged 16 to 65 by employment status, OECD average and countries, 2012

Italy

Korea

Slovak Republic

Austria

Ireland

Denmark

Estonia

United States

Canada

Norway

Australia

Japan

0 5 10 15 20 25 30

46

88

99

910

11111212

131314

15161616

1919

2124

25

Percent of workers with literacy proficiency of level 4 or 5

Percent

Italy

France

United States

Canada

Austria

OECD average

Cyprus

Flanders (Belgium)

Czech Republic

Australia

Netherlands

Finland

0 5 10 15 20 25 30

2622

191616

1514

1313131313

1212

1211

101010

99

87

5

Percent of workers with literacy proficiency of level 1 or below

Percent

Skills in the Labour Force• Contrary to what might be expected,

Canada and the United States have an almost equal proportion of working people at levels 4/5 and level 1 and below Italy’s and Korea’s performance better

fits expectations

Immigrants

Italy

Spain

Korea

United States

Germany

Norway

OECD Average

England/N. Ireland (UK)

Estonia

Ireland

Slovak Republic

200 210 220 230 240 250 260 270 280

228229

232235235

238239240241242

245247247248

255256256

260263

268268

271

Literacy mean proficiency scores offoreign-born population

Score

Ireland

Czech Republic

Australia

Spain

Canada

Austria

Germany

Flanders (Belgium)

Denmark

Norway

Finland

0 10 20 30 40 50 60

566

1013

2123232425

2629

34363737383838

4351

54

Mean score differences on the literacy scale between native- and foreign-born adults

Score point difference

Literacy — Average scores and score point differences of population aged 16 to 65 by immigrant status, OECD average and countries, 2012

Immigrants• Canada and Australia have very

diverse populations with a range of educational backgrounds, but immigration status does not factor as highly in determining overall literacy attainment in our two countries as in others

The skills of young adults

All Adults Compared with Young Adults Difference in literacy (points) between young

and all adultsDifference in numeracy (points) between young

and all adultsDifference in PS-TRE (Level 2 and above, %)

between young and all adultsKorea 20 Korea 18 Korea 33Poland 15 Spain 9 Estonia 23France 13 France 9 Flanders (Belgium) 23Spain 12 Poland 9 Czech Republic 22Estonia 11 Estonia 5 Finland 20Netherlands 11 Netherlands 5 Poland 19Italy 10 Austria 4 Austria 18Flanders (Belgium) 10 Italy 4 Germany 18Finland 9 Germany 3 Sweden 18Germany 9 Canada 3 Netherlands 17Austria 8 OECD Average 3 OECD Average 17OECD Average 7 Finland 3 Ireland 15Czech Republic 7 Flanders (Belgium) 2 Slovak Republic 15Denmark 5 Australia 2 Canada 14Ireland 4 Ireland 2 Norway 14Australia 4 Czech Republic 2 Australia 13Sweden 4 Slovak Republic 2 Denmark 12Japan 3 Cyprus 0 Japan 11Canada 2 Sweden -1 England/N. Ireland (UK) 8Slovak Republic 2 United States -3 United States 6United States 2 Japan -5 Cyprus –Cyprus -2 Denmark -5 France –Norway -3 England/N. Ireland (UK) -5 Italy –England/N. Ireland (UK) -7 Norway -7 Spain –

The skills of young adults

PS-TRE ― Proficiency of population aged 16 to 65 at Level 2 or 3 by age groups, OECD average and Canada, 2012

16 to 24 25 to 34 35 to 44 45 to 54 55 to 650

10

20

30

40

50

60

51 49

38

23

11

51 49

42

28

17

OECD average Canada

Age group

Perc

ent

Youth• Do we need to look at new ways of

knowledge transfer between older and younger workers?

• How do employment opportunities (primarily p/t vs. primarily f/t affect attainment)?

• Other questions will be forthcoming …

Where Does Canada Go Next?• A series of thematic reports

Education Official Language Minority Communities Labour Market Social Outcomes Aboriginal Populations New Canadians

• PIAAC Online

Where do WE Go Next?

Together • WE can look at building richer programs

and services for individuals, families, employees and communities

• WE can support individuals in a life long learning continuum that is reflective of their changing circumstances

• WE can honour the holistic teachings of the communities we have all worked with

www.piaac.ca

http://www.oecd.org/site/piaac/

http://piaacdataexplorer.oecd.org/ide/idepiaac/

Questions?

Brett WaytuckProvincial Librarian / Executive Director

Provincial Library & Literacy Officebrett.waytuck@gov.sk.ca