Overview of the studies III PIAAC
Andrés Sandoval-Hernández – IEA DPC
Workshop on using PISA PIAAC, TIMSS & PIRLS, TALIS datasets
Ispra, Italy- June 24-27, 2014
Note: These slides were prepared as part of the IEA training portfolio with the collaboration of IEA staff and resource persons.
Table of Contents
• Why international assessments?
• What is PIAAC?
• The assessment design
• What does the study provide?
• First steps ??
2
Why International Assessments?
• Benchmarking function
• Provide comparative analysis of skill-formation systems and their outcomes, as well as international benchmarking regarding adult skills
• Analytic function
• Suggest hypotheses about • Relationship between adult skills and factors that may influence them
• Areas where skill-formation systems have particular strengths or weaknesses
3
Table of Contents
• Why international assessments?
• What is PIAAC?
• The assessment design
• What does the study provide?
• First steps ??
4
What is PIAAC?
• Programme for the International Assessment of Adult Competencies
• Assesses proficiency of adults in four information-processing skills considered essential for successful participation in the information-rich economies and societies of the 21st century:
• Literacy
• Numeracy
• Problem solving in technology-rich environments (PSTRE)
• Reading Components
5
What is PIAAC?
• Assessment of information-processing skills
• Skills that are considered essential to effectively access, understand, analyze and use text-based information, and in some cases with mathematics, information in the form of representations of various types (e.g., pictures, graphical representations)
• Purpose is to gain a picture of the distribution of the adult population over the entire spectrum of proficiency in each of the domains assessed
6
What is PIAAC?
• These are key competencies (or skills) in that they are:
• Preconditions for successful integration and participation in the labor market, education and training, and social and civic life
• Relevant to all adults
• Highly transferable in that they are relevant to multiple social fields and work situations
• ‘Learnable’ and therefore, subject to the influence of policy
7
Table of Contents
• Why international assessments?
• What is PIAAC?
• The assessment design
• What does the study provide?
• First steps ??
8
PIAAC Survey Instruments
• Background Questionnaire
• Direct Assessment
• Module on skills use
• Computer delivered
• Paper-based option for individuals with no computer experience
9
PIAAC Background Questionnaires
• Collect a comprehensive set of information designed to support the major analytical objectives of PIAAC:
• Determine the level and the distribution of proficiency in key information processing skills for important subgroups of the adult population
• Better understand factors associated with the acquisition, development, maintenance and loss of proficiency over the life cycle
• Better understand the relationship of proficiency in information-processing skills to economic and other social outcomes
10
PIAAC Background Questionnaires
• Collect information in five main areas:
• Basic demographic characteristics and background of respondents
• Educational attainment and participation
• Labor force status and employment
• Social outcomes
• Literacy and numeracy practices and the use of skills
11
PIAAC Background Questionnaires
• Characteristics and background of respondents
• Demographics: Age, sex, country of birth
• Household and family structure: Number of persons in household, living with spouse or partner, activity of spouse/partner, number and age of children
• Language background: First and second languages spoken by the child, language currently most often spoken at home
• Immigration status: Age at which respondent immigrated, country of birth of parents
• Social background: Highest level of education of parents, number of books in home at age 16
12
PIAAC Background Questionnaires
• Educational experience and current participation in learning activities: • Educational experience: Highest qualification, in which
country it was gained, field of study, age completed
• Current study: Level of course, field of study
• Incomplete study: Level of course, age at which person left the course
• Formal studies in previous year: • Undertaken formal studies in previous year, how many courses, level
of last course, reason for undertaking study, employed while studying, study took place inside or outside working hours, usefulness of course to work, type of employer support received
13
PIAAC Background Questionnaires
• Educational experience and current participation in learning activities
• Non-formal courses in previous 12 months: Type and frequency
• Most recent non-formal activity: Type, relation with job, reason for participating, employer support
• Total time in education and training activities, proportion of time in job-related activities
• Barriers to undertaking education and training
14
PIAAC Background Questionnaires
• Labor force status, work history and job characteristics
• Current activity: Labor force status (ILO definition), main current activity
• Work history: had ever worked, had paid work in previous 12 months, age stopped working (if unemployed), total time in employment, number of employers in previous 5 years
15
PIAAC Background Questionnaires
• Labor force status, work history and job characteristics
• Current job: Industry, occupation, employee or self-employed, age started with current employer, establishment size, number of employees increasing or decreasing, part of larger organization, number of employees (if self-employed), management of supervisory responsibilities, number of subordinates, type of employment contract, usual working hours, extent of flexibility regarding job tasks, job satisfaction, gross wages or salary, (if self-employed) earnings from business
16
PIAAC Background Questionnaires
• Labor force status, work history and job characteristics
• Most recent job (if unemployed): Industry, occupation, employee or self-employed, when left last employer, establishment size, number of employees (if self employed), management of supervisory responsibilities, number of subordinates, type of employment contract, usual working hours, main reason for leaving last job
17
PIAAC Background Questionnaires
• Social participation and health
• Trust: Trust in others, perception of others, behavior towards self
• Political efficacy: Influence on political process
• Volunteering: Frequency of voluntary work in previous 12 months
• Health status: Self-assessed health status
18
Why Collect these Data?
• There are different but related reasons:
• Engagement in these domains is a constituent element of the respective measurement frameworks
• Engagement in related practices is an important correlate of proficiency in the domains
• Policy interest in gaining information on a broader range of generic skills than provided by the direct assessment
• Policy interest in better understanding the demand for skills and phenomena such as qualifications and skills match/mismatch related to the interaction of supply and demand
19
Direct Assessment
• Evaluates the skills of adults in 4 fundamental domains
• These are considered to constitute “key” information processing skills for the development of higher-order cognitive skills
• The competencies assessed are:
• Literacy
• Reading components
• Numeracy
• Problem solving in technology-rich environments
20
Direct Assessment - Literacy
• Ability to understand and use information from written texts in a variety of contexts to achieve goals and develop knowledge and potential
• This is a core requirement for developing higher-order skills and for positive economic and social outcomes
• Previous studies have shown literacy to be linked to positive outcomes at work, to social participation, and to lifelong learning
21
Direct Assessment - Reading components
• To provide more detailed information about adults with poor literacy skills, the literacy assessment in this survey is complemented by a test of “reading component” skills
• These are the basic set of decoding skills that enable individuals to extract meaning from written texts:
• knowledge of vocabulary,
• ability to process meaning at the level of the sentence, and
• fluency in reading passages of text
22
Direct Assessment - Numeracy
• Ability to use, apply, interpret, and communicate mathematical information and ideas.
• It is an essential skill in an age when individuals encounter an increasing amount of quantitative and mathematical information in their daily lives
• Numeracy is a skill parallel to reading literacy, and it is important to assess how these competencies interact
23
Direct Assessment - Problem solving in technology-rich environments
• Ability to use technology to solve problems and accomplish complex tasks
• It is a measurement of the cognitive skills required in the information age
• In this survey, higher-order skills are identified along with basic proficiency
24
Module on Skills Use
• A “job-requirements approach” to ask adults who are employed about a number of generic skills they use in the workplace
• The survey asks how intensively and how frequently they use these skills at work
• Information is also collected about four broad categories of generic work skills:
• cognitive skills,
• interaction and social skills,
• physical skills, and
• learning skills. 25
Module on Skills Use
• Cognitive skills encompass reading, writing, mathematics and the use of information and communication technologies
• Interaction and social skills cover collaboration and co-operation, planning work and use of time, communication and negotiation, and customer contact
• Physical skills involve the use of gross and fine motor skills
• Learning skills cover activities such as instructing others, learning (formally or informally), and keeping up-to-date with developments in one’s professional field
26
PIAAC Assessment Design
27
PIAAC Target Population
• Non-institutionalized population ages 16-65 residing in the country at the time of data collection, irrespective of nationality, citizenship or language status
• The normal territorial unit covered by the OECD survey of adult skills was the country as a whole
28
PIAAC Administration
• PIAAC was administered under the supervision of trained interviewers either in the respondent’s home or in a location agreed upon between the respondent and the interviewer
• Following the identification of the sampled person, the administration of the survey involved two stages
• Completion of the Background Questionnaire
• Completion of the cognitive assessment
29
Table of Contents
• Why international assessments?
• What is PIAAC?
• The assessment design
• What does the study provide?
• First steps ??
30
PIAAC Proficiency Scales
• Results from PIAAC are reported on a scale ranging between 0 and 500
• Test takers and test items are located on the scale in terms of their proficiency and their difficulty respectively
• Scores assigned to respondent were in the form of 10 plausible values
• Proficiency levels are defined for each of the domains
31
PIAAC Proficiency Levels
• Numeracy and Literacy Scale • 376: Level 5
• 326: Level 4
• 276: Level 3
• 226: Level 2
• 176: Level 1
• Problem Solving in Technology Rich Environments Scale • 341: Level 3
• 291: Level 2
• 241: Level 1
32
Table of Contents
• Why international assessments?
• What is PIAAC?
• The assessment design
• What does the study provide?
• First steps ??
33
Where do I begin?
• Read the Technical Report... Most questions you may have are already been answered there
• Review the survey instruments and country-specific materials
• Formulate research question
• You will have to make decisions that depend on: • Your interests
• Prior research
• Your computer resources
• Other resources available
• Available data
34
Where do I begin?
• You are very lucky! There are several software packages out there that you can use to analyze the data
• IDB Analyzer
• WesVar
• International Data Explorer
• IEA Data Visualizer
• Mplus, Stata, R and other software (using special routines)
• PIAAC data:
• Go to: http://piaacdataexplorer.oecd.org/ide/idepiaac/
35
In Summary
• PIAAC assessed adults in four main domains:
• Literacy
• Numeracy
• Reading component
• Problem solving in TRE
• Additionally it collects information in four categories of generic work skills:
• cognitive skills,
• interaction and social skills,
• physical skills, and
• learning skills. 36
In Summary
• This study has complex sampling and test designs
• If you do not take into account the sample and test design in your analysis you simply end up with the wrong answer
• Be careful with inferences of causality when using these data
37
Thank You!
38
Questions?
