#EATPconf
Blending Workforce Readiness Curriculum, Assessments, &
Credentials
Peter Shiras (International Youth Foundation)
Richard D. Roberts, Ph.D. (Center for Innovative Assessments, ProExam)
Overview
– Presenter Introductions
– Workforce Readiness: Background
– Impact from Policy
– Key Research Findings
– Workforce Readiness Assessments: Constructs and Methods
– Workforce Readiness Curriculum: An Overview of Passport to Success
– Workforce Readiness Assessment in an International Context: Challenges and Strategies
– Workforce Readiness Assessments in an International Context: Preliminary Findings
– Summary
About the Presenters: Peter Shiras
– International Youth Foundation
– Executive Vice President for Business Development
– Work in 70 countries
– Partnerships with major companies including Hilton Worldwide, PepsiCo, Caterpillar, Microsoft, Chevron, Walmart, and Cummins
– Passport to Success
– Signature life skills and work readiness training program
– Work with universities, TVET systems and companies
– Developed with support from GE Foundation
– Significant impact on work readiness, job placement, retention and advancement
About the Presenters: Richard D. Roberts
– NRC Fellow (USAF)
– University of Sydney
– ETS
– Background variables: PISA, PIAAC, OECD
– Cross-cultural competence (3C)
– Fake resistant assessments of noncognitive skills
– ProExam
– Assessment of Biases in Cognition (IARPA)
– Assessment of non-academic skills (Tessera)
– Develop innovative assessments, feedback, & action plans
– Books, published articles on workforce readiness, noncognitive (social and emotional) skills
#EATPconf
Workforce Readiness: Background
The Changing Nature of Work (Burrus et al., 2016)
6
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Technological Innovation
The rate of technological innovation is increasing
• Internet subscriptions nearly tripled in OECD countries from 1999-2009 (OECD, 2013)
• Mobile phone subscriptions more than tripled OECD countries from 1999-2009 (OECD, 2013)
Work Activity ShiftTechnological innovation leads to increased technology use in day-to-day work activities
• Over 95%, 85%, and 65% of large, medium, and small business, respectively, in OECD countries now use the internet in their jobs (OECD, 2013)
• Increase in non-routine analytic and interpersonal job tasks performed by workers since 1960 (Autor & Price, 2013)
Job Market ShiftShift in work activities leads to a shift in the job market
• Growth in occupations needing highly educated workers and decline in occupations needing medium- and low-educated workers from 1998-2008 (OECD, 2013)
• 47% of U.S. jobs at high risk due to automation over the coming decades, with low skill and low paying jobs most likely to be automated (Frey & Osborne, 2015)
Skill ShiftShift in the job market requires a shift in worker skills emphasized
• Most job growth since 1980 has been in occupations that require strong social (Deming, 2015).
• Jobs that require high levels of analytical and mathematical reasoning have had poor job growth since 1980 (Deming, 2015)
Valued by The Workforce (Conference Board, 2005)
7
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
50
60
70
80
90
100
Req
uir
e Th
is S
kill
(%
of
Emp
loye
rs)
Mismatch Between Job Candidates and Employers
8
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
(Source: Hart Research Associates, 2015)
How Well-Prepared is Current Workforce on this Skill? Employers Students
Working in teams 37% 64%
Staying current on technologies 37% 46%
Ethical judgment and decision making 30% 62%
Locating, organizing, and evaluating information 29% 64%
Oral communication 28% 62%
Working with numbers/statistics 28% 55%
Written communication 27% 65%
Critical/analytical thinking 26% 66%
Being innovative/creative 25% 57%
Analyzing/solving complex problems 24% 59%
Applying knowledge/skills to the real world 23% 59%
Awareness/experience of diverse cultures 21% 48%
Valued by The Workforce (Sackett & Walmsley, 2015)
9
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
O*NET Worker Style CharacteristicJob Family Top Ranked Second Ranked Third RankedArchitecture and Engineering Analytical Thinking (O)/ Dependability (C) Integrity (C) Initiative (C)
Arts, Design, Entertainment, Sports, and Media
Dependability (C) Adaptability and Flexibility (N) Initiative (C)/ Stress Tolerance (N)
Building and Grounds Cleaning and Maintenance
Dependability (C) Cooperation (A) Self-Control (N)
Business and Financial Operations Integrity (C) Dependability (C) Analytical Thinking (O)/ Cooperation (A)
Computer and Mathematical Analytical Thinking (O) Dependability (C) Cooperation (A)
Construction and Extraction Dependability (C) Cooperation (A) Self-Control (N)
Education Dependability (C) Integrity (C) Self-Control (N)
Farming, Fishing, and Forestry Dependability (C) Self-Control (N) Independence (C)
Food, Preparation, and Serving Related Cooperation (A) Dependability (C) Self-Control (N)
Health Care Support Dependability (C) Concern for Others (A) Integrity (C)
Health Care Practitioners and Technical Integrity (C) Dependability (C) Concern for Others (A)
Installation, Maintenance, and Repair Dependability (C) Integrity (C) Cooperation (A)
Legal Integrity (C) Dependability (C) Analytical Thinking (O)
Life, Physical, and Social Science Integrity (C) Analytical Thinking (O) Dependability (C)
Management Dependability (C) Integrity (C) Leadership (E, A)Office and Administrative Support Dependability (C) Integrity (C) Cooperation (A)
Personal Care and Service Dependability (C) Self-Control (N) Integrity (C)
Production Dependability (C) Cooperation (A) Integrity (C)Sales and Related Dependability (C) Integrity (C) Self-Control (N)Transportation and Material Moving Dependability (C) Self-Control (N) Integrity (C)
Predictive of Workplace Performance
10
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Job
Performance
Task
Performance
Organizational
Citizenship
Behavior
Counter-Productive
Work Behavior
Conscientiousness .33 .31 .40 -.40
Agreeableness .22 .13 .23 -.51
Emotional Stability .13 .11 .21 -.31
Openness .10 .14 .04 -.08
Extraversion .26 .15 .28 -.04
(Source: Roberts et al., 2015)
Noncognitive Skills Impact Long-Term Workforce Status
• Swedish enlistment data: Measured at age 18 and predict labor market outcomes for 30+ year olds
• Noncognitive skills measured
– Willingness to assume responsibility, independence, outgoing character, persistence, emotional stability, initiative, social skills
• Poor labor market outcomes (chronic unemployment, low earnings)
– Predicted by lack of noncognitive skills rather than cognitive ability
– Noncognitive skills determine productivity irrespective of occupation or ability level. Cognitive ability of value for those in highly skilled occupations (but this may be shifting)
(Source: Lindqvist & Vestman, 2011)
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Noncognitive Skills And Change
12
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Effe
ct S
ize
Roberts, B. W., Walton, K. E. & Viechtbauer, W. (2006a). Patterns of mean-level change in personality traits across the life course: A meta-analysis of longitudinal studies. Psychological Bulletin, 132, 1–25.
Leadership
0
0.3
0.6
0.9
1.2
10 to 18 18 to 22 22 to 30 30 to 40 40 to 50 50+ 60-70 70+
Social dominance Agreeableness Conscientiousness
Emotional Stability Openness
Leadership
Key Take Away Messages
• Emerging consensus from the world of policy suggests:
– Modern workforce may require noncognitive skills in equal measure to cognitive skills
– But generally not the focus of education
– There is a mismatch in perceptions of competence: Students => Employers
• Emerging research suggests:
– Equally predictive as cognitive tests for certain outcomes (e.g., job ratings, training time), better for others (e.g., attrition)
– Can actually be measured with precision (e.g., reliability, validity)
– Noncognitive skills are malleable (cf. cognitive constructs)
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
#EATPconf
Workforce Readiness Assessment: Constructs and
Methods
Competency Crosswalks I
15
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Competency Big FiveIYF (PTS Lessons) O*NET
Child Trends
Dependability/ Responsibility
Facet of Conscientiousness
Responsibility
Workplace Protocol
Dependability (Conscientiousness, Work Style)
Self-Control, Higher-Order Thinking Skills
Work Ethic Facets of Conscientiousness and EmotionalStability
Goal Setting
Stress Management
Managing Emotions
Achievement/Effort (Achievement Orientation & Stress ToleranceWork Styles)
Positive Self-Concept, Self-Control, Higher-Order Thinking Skills
Competency Crosswalks II
16
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Competency Big FiveIYF (PTS Lessons) O*NET
Child Trends
Time Management
Facet of Conscientiousness
Time Management
Time Management (Resource Management Skills, Conscientiousness, Work Style)
Self-Control
Integrity Facets of multiple Big Five constructs – especially Agreeableness (see also HEXACO model)
Keys to Being a Good Employee
Integrity (Conscientiousness Work Style)
Self-Control
Competency Crosswalks III
17
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Competency Big Five IYF (PTS Lessons) O*NETChild Trends
Cooperation/ Team Work
Facet of Agreeableness
Team Player
Power of Positive Attitude
Managing Conflict
Respecting Authority
Respecting Diversity
Cooperation (facet of Interpersonal Orientation Work Style )
Social Skills, Communication Skills
Critical Thinking/Problem-Solving
Facet of Openness
Solving Problems
Creative Thinking
Analytical Thinking (facet of Practical Intelligence Work Style)
Higher-Order Thinking Skills
Traditional Likert Scale Items
18
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved. (Source: PISA Background Variable Questionnaire 2012)
ST46
Thinking about the mathematics you do for school: to what extent do you agree with the following statements?
Strongly agree Agree Disagree
Strongly disagree
I finish my homework in time for mathematics class. 1 2 3 4
I work hard on my mathematics homework. 1 2 3 4
I am prepared for my mathematics exams. 1 2 3 4
I study hard for mathematics quizzes. 1 2 3 4
I keep studying until I understand mathematics material. 1 2 3 4
I pay attention in mathematics class. 1 2 3 4
Representative Finding
• Time Management: Determining one’s needs, setting goals to meet needs, and prioritizing and planning to meet goals (Lakein, 1973)
• Time Management typically relates to GPA around the same magnitude as cognitive test scores
Planning Organization
Grades .21** .38**
Grades (6 mo later) .25** .42**
(Source: Liu, Rijmen, MacCann, & Roberts, 2009)
• And evidence-based interventions of time management exist, including elements that have been infused into school curricula (Burrus, Jackson, Holtzman, & Roberts, 2013)
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
The Little Problems of Likert Scales
20
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
– Faking
– Self-insight
– Above Average Effects
– Outcome evaluation
– Reference Bias
– Cross-cultural comparison ST84
Q Below you will find descriptions of three mathematics teachers. Read each of the descriptions of these teachers, then let us know to what extent you agree with the final statement.
(Please tick only one box in each row.)
Strongly
agree Agree Disagree Strongly
disagree
a) The students’ in Ms. <name’s> class
frequently interrupt her lessons. She
always arrives five minutes early to
class. Ms. <name> is in control of her
classroom.
1 2 3 4
b) The students’ in Ms. <name’s> class
are calm and orderly. She always
arrives on time to class. Ms. <name> is
in control of her classroom. 1 2 3 4
c) The students’ in Mr. <name’s> class
frequently interrupt his lessons. As a
result, he often arrives five minutes
late to class. Mr. <name> is in control
of his classroom.
1 2 3 4
ST85
Q Thinking about the mathematics teacher who taught your last mathematics class: to what extent do you agree with the following statements?
(Please tick only one box in each row.)
Strongly
agree Agree DisagreeStrongly disagree
a) My teacher gets students to listen to him
or her. 1 2 3 4
b) My teacher keeps the class orderly. 1 2 3 4
c) My teacher starts lessons on time. 1 2 3 4
d) The teacher has to wait a long time for
students to <quiet down>. 1 2 3 4
Vignettes Self-report
Forced-Choice Assessment
For each pair of statements, please click on the one that is most like you.
1. I respect my boss
2. I am always prepared
• New development: Can use psychometric modeling to recover both ipsative and normative scores
• Representative finding: Forced-choice had similar validity coefficients as Likert-scale for predicting valued educational outcomes (Anguiano-Carrasco, MacCann, Geiger, Seybert, & Roberts, in press)
• Advantage: This method appears less fakable, even in high stakes situations (Zigler, MacCann, & Roberts, 2011)
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Situational Judgment Test
DEPENDABILITYYour job involves work for multiple different clients. You have a deliverable due in 6 weeks, and your manager has estimated that this will probably take 2 weeks of your time to complete. You do not have anything urgent due this week.
1. Start preparing the client deliverable right away.2. Start preparing the client deliverable 3 weeks before the due date.3. Start preparing the client deliverable 2 weeks before the due date.4. Start preparing the client deliverable 1 week before the due date.
• Suggestive finding: Correlates over .40 with similar self-report constructs, and predicts valued outcomes (e.g., sales performance, GPA) often more highly than “flat” Likert scales (Roberts et al., 2016a, 2016b)
• Can be used as both a summative and formative assessment approach
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Situational Judgment Test (Continued)
Managing EmotionsAnna has been overseas for a long time and returns to visit her family. So much has changed that Anna feels left out. What action would be the most effective for Anna?
a. Nothing, it will sort itself out soon enoughb. Tell her family she feels left outc. Spend time listening and getting involved againd. Reflect that relationships can change over time
Construct Correlation (raw) with STEM
TAS-20 (Externally-Oriented Thinking) -.36** (-.32**)
Psychology Grade (Actual) .26** (.18)
(MacCann & Roberts, 2008)
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Sample Individual Reports
24Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Sample Institutional Reports
25Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
#EATPconf
Workforce Readiness Curriculum: An Overview Of
Passport to Success
#EATPconf
Best practice curriculum
©2016. All Rights Reserved. International Youth Foundation.
Original Development Partner: General Electric (GE) Foundation
Goal: To equip young people, ages 14-29, with key skills to help them stay in school, find employment, and succeed in life
Key features:
• Implemented in 40+ countries and
utilized in 19 languages
• Implemented in different settings: on the
job training, secondary schools, post-secondary schools,
vocational/technical colleges, and teacher training programs
• More than 3,300 IYF-trained trainers
• More than 113,000 young people trained
#EATPconf
Key course elements
■ A 78-module curriculum with success at contextualizing content to specific contexts (i.e. cultural setting; youth demographic; employment sectors; etc.).
■ Trainer’s Guide that supports trainers to deliver PTS lessons with consistency and quality.
■ Trainee Handbook that supports students as they progress through the course.
■ Comprehensive Training of Trainers and Master Trainers program that equips trainers through multi-day, interactive workshops with the skills and confidence necessary to successfully deliver experiential, student-centered methodology.
■ Coaching resources to support ongoing guidance and support to trainers.
■ Certification of trainers and coaches to implement PTS program in their institutions based on IYF certification standards and protocol.
■ Tracking and Performance Measurement tools including pre- and post- tests measuring skills and behavior change, teacher and student focus groups and employment and educational outcomes tracking mechanisms.
■ Quality assurance standards to support effective and impactful programming.
©2016. All Rights Reserved. International Youth Foundation.
#EATPconf
COMPREHENSIVE CURRICULUM
©2016. All Rights Reserved. International Youth Foundation.
Core Life Skills
• Self-confidence
• Managing Emotions
• Personal responsibility
• Respecting self and others
• Cooperation/Team work
• Communication/Inter-personal skills
• Creative Thinking
• Critical Thinking/Problem solving
• Decision making
• Conflict management
Curriculum Structure
80 Lessons / 6 units available:
• Personal Competencies
• Problem Solving & Managing Conflict
• Healthy Behaviors
• Job Search & Effective Work Habits
• Skills for Professional Growth/ Entrepreneurship
• Service Learning
#EATPconf
PTS speaks to a youth audience, while enhancing teachers’ training styles
PTS provides different learning experiences and suits different learning styles (auditory, visual, kinesthetic)
Generating Interest in Topic
Information to Share
Small and/or Large Group Activity
Youth learn to apply lessons outside the classroom
Personal Application
Pedagogy & methodology
©2016. All Rights Reserved. International Youth Foundation.
#EATPconf
Select global results
■ Jordan: 90% of employers say they would hire more graduates in the future 80% of employers surveyed report that PTS graduates meet their company’s needs.
■ Egypt: 75% of employers stated there is a remarkable and noticeable difference between PTS graduates and other young employees.
■ South Africa: 91% of young Hilton Team members who received PTS in South Africa reported being more thoughtful listeners; 75%indicated improvement in conflict management and cooperation skills.
■ Caribbean: 90% of employers in the Caribbean hiring PTS graduates said they would do so again.
©2016. All Rights Reserved. International Youth Foundation.
#EATPconf
Workforce Readiness in an International Context:
Challenges and Strategies
Assessment Uses
Possible uses include:
– Training
– Outcomes assessment (evaluating efficacy of Passport to Success program)
– Self-driven professional development and enhancement
– International comparison (cf. OECD)
– Issuing a micro-credential (badge)
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Legal Issues Related to Assessment Vary by Country
Country
Evidence Needed to
Refute a Claim
Consequences of
Violation
Permissibility of Preferential
TreatmentChile Unclear, unless for sexual
harassment or unionization
suits. Empirical evidence not
required
Unknown. Currently, sexual
harassment suits may result in
monetary compensation and up
to 3 years imprisonment.
Government has enacted an informal quota
for women in minister positions; however,
this has not crossed over into the private
sector.
Kenya Must show that decisions
were based on applicant
aptitudes and abilities.
Empirical validity evidence
not required.
Remedy by following
recommendations of Ministry of
Health, Labor, and Welfare.
Possible public announcement of
violation. Civil fine of maximum
¥200,000
Different cutoff scores are set for members
from different ethnic groups to ensure that
some members from each group will be
selected. There are required quotas of 5% in
both the private and the public sector for
disabled individuals.
South Africa Both qualitative and
empirical data can be brought
to bear to support the
validity.
Fines. Possible cancellation of
government contracts.
Preferential treatment is permitted and
applied. Racial quotas are legal and
practiced by many large employers. The
practical implication for this is that it is legal
in South African context to use race
norming, or within-group top-down
selection strategies, in order to address
affirmative-action needs of organizations.
(Source: Adapted from Myors et al., 2008)Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Translation Issues & Solutions
– Social and cultural mores vary, so writing SJTs is a complex undertaking (e.g., party scenario inappropriate in Saudi Arabia)
– Key guidelines for writing noncognitive items follow emerging guidelines from various sources (e.g., OECD): Avoid negations, idioms, use <“radicals”> to replace names, monetary units etc in different countries
– What is now standard fare:
– Forward and backward translations
– Local cognitive labs
– DIF analysis
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Validation Framework
#EATPconf
Workforce Readiness in an International Context:
Preliminary Findings
Pilot Study: Item Try-Out
– Small bilingual sample from South Africa (N =20, F = 11, Age = 21 to 31)
– Situational judgment (8 items)
– 6 items had acceptable properties, scores ranged between .55 and .85 (Mean = .69)
– Self-report (24 items)
– Mean 4.3 (mean SD = .75) on a 1 to 5 scale
– This suggests high degree of social desireablity in these items
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
Pilot Study: Usability
Test-Specific Feedback Results from South Africa Cognitive Laboratory:
Percentage of Participants Who Responded “Agree“ or “Strongly Agree”
General Test-Specific Feedback Questions
Situational
Judgment
Test
Self-Report
Personality
I enjoyed responding to these items more than responding to
the questions on most tests I have taken. 70 85
These items seemed more relevant to my day-to-day life than
the questions on most tests I have taken. 70 85
In general, I enjoy taking tests that help me to learn more
about my personal qualities. 100 100
I would like to take another test like this in the future. 90 100
I would gladly take another test like this if doing it meant I got
feedback about my personal qualities. 90 100
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.
#EATPconf
Summary
#EATPconf
Next Steps
– Item pool being expanded
– Translation protocols to be followed (note, extra burden here regarding score reports, which also must be translated)
– Pilot studies in Mexico, Indonesia, and South Africa
– Studies to establish cut-scores
– Possible use in evaluation studies
– Local validation studies
– Delivery: IT in each country
– Marketing and branding
Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.