Hard Evidence on Soft Skills - University of Chicago · 2012-03-29 · Hard Evidence on Soft Skills...

Introduction Defining/Measuring Causal Evidence Summary References

Hard Evidence on Soft Skills

James J. Heckman and Tim Kautz

IRP Lampman LectureMadison, WIMay 16, 2012

This draft, March 29, 2012

Heckman and Kautz Hard Evidence on Soft Skills 1 / 72


Introduction

Widespread use of standardized achievement tests.

But the traits that they measure are not well-understood.

This paper summarizes recent evidence ona what achievement tests capture and what they do not;b how achievement tests relate to other measures of “cognitive

ability” like IQ and grades;c the important skills that achievement tests miss or

mismeasure, andd how much these other skills matter in life.



Achievement tests miss, or perhaps more accurately, do notadequately capture, soft skills—personality traits, goals,motivations, and preferences that are valued in the labormarket, in school, and in many other domains.

The larger message of this research is that soft skills predictsuccess in life, that they causally produce that success, andthat programs that enhance soft skills have an important placein an effective portfolio of public policies.



Standardized achievement tests: designed to capture “generalknowledge” produced in schools and through life experiences.

Such knowledge thought to be relevant to success inside andoutside of the classroom.

However, achievement tests are often validated using otherstandardized achievement tests or other measures of cognitiveability.

This practice is inherently circular.



A more relevant validity criterion is how well these tests predictmeaningful outcomes, such as educational attainment, labormarket success, crime, and health.



Findings from Recent Research on Personality andEconomics

Success in life depends on personality traits that are not wellcaptured by measures of cognition.

Conscientiousness, perseverance, sociability, and curiositymatter.

While economists have largely ignored these traits, personalitypsychologists have studied them over the last century.



Do Stable Traits Exist?

Many scholars—inside and outside of psychology—havequestioned the existence of stable personality traits, arguingthat constraints and incentives in situations almost entirelydetermine behavior.

A substantial body of evidence shows that stable traits exist.

People tend to behave in the same fashion across a wide rangeof situations.

Evidence from genetics and neuroscience provides a biologicalbasis for the existence of such traits, suggesting that somethingtied to the person, not the just the situation, affects behavior.



Identification Problems: How are Personality TraitsMeasured?

Psychological traits are not directly observed.

No ruler for perseverance, no caliper for intelligence.

All cognitive and personality traits are measured usingperformance on “tasks,” broadly defined.

Different tasks require different traits.

Some distinguish between measurements of traits andmeasurements of outcomes, but this distinction is misleading.

Both traits and outcomes are measured using performance onsome task.



Psychologists sometimes claim to circumvent this measurementissue by creating taxonomies of traits and by applying intuitivenames to responses on questionnaires.

They are still rooted in task performance or behavior.

Responding to a questionnaire is a task.



Determinants of Task Performance

Performance in most tasks depends on preferences, personalitytraits, cognitive ability, and incentives although the importanceof each differs by task.

This dependence creates the fundamental identificationproblem in measuring traits.

Measured cognitive ability and measured personality depend ona constellation of factors.

Multiple traits affect performance on cognitive tasks.



Predictive Power of These Measures is High

Despite these qualifications, measures of personality traitspredict meaningful life outcomes.

Conscientiousness – the tendency to be organized, responsible,and hardworking—is the most widely predictive of thecommonly used personality measures.



Causal Status of Measured Traits

Most studies in psychology only report correlations betweenmeasured traits and outcomes without addressing whether thetraits cause the outcomes and without controlling for the othertraits and incentives that determine performance on the tasksused to measure the traits.

While traits are relatively stable across situations, they are notset in stone.

On average, Agreeableness and Conscientiousness tend to growwith age.



Establishing Causality of the Traits

First, we show how an achievement test, the GeneralEducational Development (GED) test, fails to captureimportant traits that affect success in life.

Second, we show how an early childhood intervention, thePerry Preschool Program, improved the lives of disadvantagedchildren, even though the program did not permanently changethe IQ of its participants.



Defining and Measuring Personality Traits:

History and Measurement of Cognitive Ability

IQ scores are widely interpreted as measuring a stable trait.

Achievement tests are typically validated using otherachievement tests, IQ tests, and grades, rather than with tasksor outcomes that matter.



Table 1: Cognitive Ability Validities

Test Validation Domain Estimate(s) Source(s)

SAT (Achievement) 1st Year College GPA 0.35 - 0.53 Kobrin et al. (2008)

ACT (Achievement) Early College GPA 0.42 ACT, Inc. (2007)

GED (Achievement) HS Senior GPA 0.33 - 0.49 GED Testing Service (2009)

DAT (Achievement) College GPA 0.13 - 0.62† Omizo (1980)

AFQT (Achievement) 9th Grade GPA 0.54 Borghans et al. (2011)

WAIS (IQ) College GPA 0.38 - 0.43 Feingold (1982)

WAIS (IQ) HS GPA 0.62 Feingold (1982)

Various IQ∗∗ 9th Grade GPA 0.42 Borghans et al. (2011)

WISC (IQ) WRAT (Achievement) 0.44 - 0.75‡ Hartlage and Steele (1977)

WISC-R (IQ) WRAT (Achievement) 0.35 - 0.76‡ Hartlage and Steele (1977)

Various IQ∗∗ AFQT (Achievement) 0.65 Borghans et al. (2011)

Stanford Binet (IQ) WISC-R (IQ) 0.77 - 0.87 Rothlisberg (1987), Greene et al. (1990)

Raven’s (IQ) WAIS-R (IQ) 0.74 - 0.84 O’Leary et al. (1991)

WIAT (Achievement) CAT/2 (Achievement) 0.69 - 0.83∗ Michalko and Saklofske (1996)



Definitions: WISC – Wechsler Intelligence Scale for Children, WISC-R – Wechsler IntelligenceScale for Children - Revised, WAIS - Wechsler Adult Intelligence Scale, Raven’s IQ – Raven’sStandard Progressive Matrices, GED – General Educational Development, DAT – DifferentialAptitude Test, WIAT – Wechsler Individual Achievement Test, CAT – California AchievementTest, WRAT – Wide Range Achievement Test† Large range is due to varying validity of eight subtests of DAT‡ Ranges are given because correlations vary by academic subject∗ Ranges are given because correlations vary by grade level∗∗ IQ test scores in the NLSY79 are pooled across several IQ tests using IQ percentiles



Validating cognitive ability tests using other measures ofcognitive ability is inherently circular.

A more relevant measure is how these tests predict outcomesthat matter.



Table 2: Predictive Validities in Outcomes that Matter

(Adjusted R-Squared)IQ Sample AFQT Sample GPA Sample

Males IQ Pers Both AFQT Pers Both GPA Pers Both

Earnings at Age 35 0.07 0.05 0.09 0.17 0.07 0.18 0.09 0.06 0.12Hourly Wage at Age 35 0.07 0.03 0.08 0.13 0.06 0.14 0.07 0.06 0.09Hours Worked at Age 35 0.01 0.03 0.04 0.03 0.02 0.03 0.02 0.01 0.02Jail by Age 35 0.03 0.02 0.04 0.06 0.06 0.09 0.03 0.03 0.04Welfare at Age 35 0.01 0.00 0.01 0.03 0.01 0.03 0.01 0.00 0.01Married at Age 35 0.01 0.05 0.05 0.04 0.03 0.06 0.03 0.03 0.04BA Degree by Age 35 0.12 0.08 0.16 0.19 0.10 0.22 0.14 0.10 0.18Depression in 1992 0.01 0.05 0.05 0.04 0.04 0.06 0.02 0.04 0.04

Adj, R2 Cog, Personality 0.07 0.17 0.11

IQ Sample AFQT Sample GPA Sample

Females IQ Pers Both AFQT Pers Both GPA Pers Both

Earnings at Age 35 0.01 0.03 0.03 0.09 0.05 0.11 0.05 0.04 0.07Hourly Wage at Age 35 0.05 0.03 0.06 0.12 0.05 0.14 0.06 0.04 0.08Hours Worked at Age 35 -0.00 0.02 0.02 0.00 0.01 0.00 0.00 0.01 0.01Jail by Age 35 -0.00 0.01 0.00 0.01 0.02 0.02 0.01 0.01 0.02Welfare at Age 35 0.02 0.04 0.05 0.10 0.05 0.12 0.05 0.05 0.07Married at Age 35 0.03 0.03 0.05 0.05 0.04 0.07 0.03 0.03 0.05BA Degree by Age 35 0.10 0.08 0.14 0.17 0.09 0.20 0.10 0.08 0.13Depression in 1992 0.02 0.05 0.05 0.04 0.05 0.07 0.02 0.05 0.05




Source: National Longitudinal Survey of Youth 1979. Table Description: The table shows the adjusted R-squared fromregressions of later-life outcomes on measures of personality and cognition. For each cognitive measure, the first columnshows the explained variance using only the measures of cognitive ability, the second column shows the explained variancefrom using only the measure of personality (Personality), and the third column shows the explained variance from using boththe measures of personality and cognition (Both). The last row shows the adjusted R-squared from a regression of eachcognitive measure on the personality measures. Measures of Personality and Cognition: The measures of personality includeminor illegal activity in 1979 (vandalism, shoplifting, petty theft, fraud and fencing), major illegal activity in 1979 (auto theft,breaking/entering private property, grand theft), participation in violent crime in 1979 (fighting, assault and aggravatedassault), tried marijuana before age 15, daily smoking before age 15, regular drinking before age 15 and any intercourse beforeage 15. It also includes measures of Self-Esteem and Locus of Control. Self-Esteem is measured using the ten-item Rosenbergscale administered in 1980. Locus of control is a measure of how much control an individual believes they have over their lifeand is measured using the 4-item Rotter scale. IQ and grades are from high school transcripts. IQ is pooled across several IQtests using IQ percentiles. GPA is the individual’s core-subject GPA from 9th grade. Outcomes: Due to the biennial nature ofthe survey after 1994, some respondents are not interviewed at age 35, for these individuals age 36 is used. Earnings includeszero-earners and excludes observations over $200,000 (2005 dollars). Hourly wage excludes observations less than $3 or over$200 (2005 dollars). Hours worked excludes observations less than 80 or more than 4000. Jail by age 35 indicates whether therespondent had listed residing in a jail or prison at some point before age 35. Welfare at age 35 indicates whether therespondent received any positive amount of welfare at age 35. Married at age 35 indicates whether the responded wascurrently married. BA degree by age 35 indicates whether the respondent received a BA degree (or higher) by age 35.Depression in 1992 is based on the 7-item Center for Epidemiologic Studies Depression Scale (CES-D). Sample: The sampleexcludes the military over sample. The samples differ across the IQ, AFQT, and GPA due to missing measures across thesamples.



Personality Measures

Personality traits are manifested through thoughts, feelings,and behaviors, and therefore, must be inferred indirectly bysome measure of performance on “tasks,” broadly defined.



Personality psychologists primarily measure personality traitsusing self-reported surveys.

They have arrived at a relatively well-accepted taxonomy oftraits called the “Big Five,” which include Openness toExperience, Conscientiousness, Extraversion, Agreeableness,and Neuroticism.



Table 3: The Big Five Domains and Their Facets

Big Five PersonalityFactor

American Psychol-ogy AssociationDictionary descrip-tion

Facets (and corre-lated trait adjective)

Related Traits Childhood Tempera-

ment Traits

Conscientiousness “the tendency to beorganized, responsi-ble, and hardwork-ing”

Competence (ef-ficient), Order(organized), Duti-fulness (not care-less), Achievementstriving (ambitious),Self-discipline (notlazy), and Delibera-tion (not impulsive)

Grit, Persever-ance, Delay ofgratification,Impulse control,Achievementstriving, Ambi-tion, and Workethic

Attention/(lackof) distractibility,Effortful control, Im-pulse control/delayof gratification, Per-sistence, Activity∗

Openness to Experi-ence

“the tendency tobe open to newaesthetic, cultural,or intellectual expe-riences”

Fantasy (imagi-native), Aesthetic(artistic), Feelings(excitable), Actions(wide interests),Ideas (curious), andValues (unconven-tional)

Sensory sensitivity,Pleasure in low-intensity activities,Curiosity








ment Traits

Extraversion “an orientation ofone’s interests andenergies toward theouter world of peo-ple and things ratherthan the inner worldof subjective expe-rience; characterizedby positive affectand sociability”

Warmth (friendly),Gregariousness(sociable), As-sertiveness (self-confident), Activity(energetic), Ex-citement seeking(adventurous), andPositive emotions(enthusiastic)

Surgency, Socialdominance, Socialvitality, Sensationseeking, Shyness*,Activity*, Posi-tive emotionality,and Sociabil-ity/affiliation

Agreeableness “the tendency to actin a cooperative, un-selfish manner”

Trust (forgiving),Straight-forwardness(not demanding),Altruism (warm),Compliance (notstubborn), Modesty(not show-off), andTender-mindedness(sympathetic)

Empathy, Per-spective taking,Cooperation,and Competi-tiveness

Irritability∗, Aggres-siveness, and Will-fulness








ment Traits

Neuroticism/ Emo-tional Stability

Emotional stabilityis “predictabilityand consistencyin emotional reac-tions, with absenceof rapid moodchanges.” Neu-roticism is “achronic level ofemotional instabil-ity and pronenessto psychologicaldistress.”

Anxiety (worrying),Hostility (irrita-ble), Depression(not contented),Self-consciousness(shy), Impulsiveness(moody), Vulnera-bility to stress (notself-confident)

Internal vs.External, Locusof control, Coreself-evaluation,Self-esteem,Self-efficacy,Optimism, andAxis I psy-chopathologies(mental disor-ders) includingdepressionand anxietydisorders

Fearfulness/behavioralinhibition,Shyness∗,Irritability∗, Frus-tration (Lack of)soothability, Sad-ness

Notes: Facets specified by the NEO-PI-R personality inventory (Costa and McCrae, 1992).Trait adjectives in parentheses from the Adjective Check List (Gough and Heilbrun, 1983).∗These temperament traits may be related to two Big Five factors.Source: Table adapted from John and Srivastava (1999).



Measured Traits May be the Manifestation of DeeperUnderlying Goals

A deeper issue, as yet not systematically investigated in theliterature in economics or psychology, is whether the traitscaptured by the alternative measurement systems are theexpression of a deeper set of preferences or goals.

Achieving certain goals requires certain traits, e.g., a surgeonhas to be conscientious and intelligent; a salesman has to beoutgoing and engaging and so forth, etc.



Identification Problems in Measuring Traits

To infer traits from behaviors requires standardizing for all ofthe other contributing factors that produce the observedbehavior.



There are two primary issues.

First, behavior depends on incentives created by situations.

Different incentives elicit different amounts of effort on thetasks used to measure traits.

Second, behavior in one task can depend on multiple traits.Not standardizing for incentives and other traits can producemisleading estimates of any trait.

These identification problems are empirically important whenmeasuring any given trait.

Examples: IQ and Effort



Figures 1 and 2 show how the variance in the scores on twoachievement tests.



Figure 1: Decomposing Achievement Tests and Grades into IQ andPersonality [NLSY79]

0.48

0.23

0.43

0.190.16

0.10

0.00

0.10

0.20

0.30

0.40

0.50

0.60

R-Sq

uare

d

IQ, Rosenberg, and Rotter IQ Rosenberg and Rotter

AFQT Grades

Achievement Grades

Source: Borghans et al. (2011). Notes: Rotter was administered 1979. The ASVAB and Rosenberg were administered in1980. AFQT is constructed from the Arithmetic Reasoning, Word Knowledge, Mathematical Knowledge, and ParagraphComprehension ASVAB subtests. IQ and GPA are from high school transcript data. AFQT, Rosenberg, and Rotter have beenadjusted for schooling at the time of the test conditional on final schooling, as laid out in Hansen et al. (2004). IQ is pooledacross several IQ tests using IQ percentiles. GPA is the individual’s core subject GPA from 9th grade. Sample excludes themilitary over-sample.



Figure 2: Decomposing Achievement Tests and Grades into IQ andPersonality [Stella Maris]

Source: Borghans et al. (2011).



The Predictive Power of Personality

Table 2 shows that personality traits predict many later-lifeoutcomes as strongly as measures of cognitive ability.



Table 2: Predictive Validities in Outcomes that Matter

(Adjusted R-Squared)IQ Sample AFQT Sample GPA Sample

Males IQ Pers Both AFQT Pers Both GPA Pers Both

Earnings at Age 35 0.07 0.05 0.09 0.17 0.07 0.18 0.09 0.06 0.12Hourly Wage at Age 35 0.07 0.03 0.08 0.13 0.06 0.14 0.07 0.06 0.09Hours Worked at Age 35 0.01 0.03 0.04 0.03 0.02 0.03 0.02 0.01 0.02Jail by Age 35 0.03 0.02 0.04 0.06 0.06 0.09 0.03 0.03 0.04Welfare at Age 35 0.01 0.00 0.01 0.03 0.01 0.03 0.01 0.00 0.01Married at Age 35 0.01 0.05 0.05 0.04 0.03 0.06 0.03 0.03 0.04BA Degree by Age 35 0.12 0.08 0.16 0.19 0.10 0.22 0.14 0.10 0.18Depression in 1992 0.01 0.05 0.05 0.04 0.04 0.06 0.02 0.04 0.04


IQ Sample AFQT Sample GPA Sample

Females IQ Pers Both AFQT Pers Both GPA Pers Both

Earnings at Age 35 0.01 0.03 0.03 0.09 0.05 0.11 0.05 0.04 0.07Hourly Wage at Age 35 0.05 0.03 0.06 0.12 0.05 0.14 0.06 0.04 0.08Hours Worked at Age 35 -0.00 0.02 0.02 0.00 0.01 0.00 0.00 0.01 0.01Jail by Age 35 -0.00 0.01 0.00 0.01 0.02 0.02 0.01 0.01 0.02Welfare at Age 35 0.02 0.04 0.05 0.10 0.05 0.12 0.05 0.05 0.07Married at Age 35 0.03 0.03 0.05 0.05 0.04 0.07 0.03 0.03 0.05BA Degree by Age 35 0.10 0.08 0.14 0.17 0.09 0.20 0.10 0.08 0.13Depression in 1992 0.02 0.05 0.05 0.04 0.05 0.07 0.02 0.05 0.05




Source: National Longitudinal Survey of Youth 1979. Table Description: The table shows the adjusted R-squared fromregressions of later-life outcomes on measures of personality and cognition. For each cognitive measure, the first columnshows the explained variance using only the measures of cognitive ability, the second column shows the explained variancefrom using only the measure of personality (Personality), and the third column shows the explained variance from using boththe measures of personality and cognition (Both). The last row shows the adjusted R-squared from a regression of eachcognitive measure on the personality measures. Measures of Personality and Cognition: The measures of personality includeminor illegal activity in 1979 (vandalism, shoplifting, petty theft, fraud and fencing), major illegal activity in 1979 (auto theft,breaking/entering private property, grand theft), participation in violent crime in 1979 (fighting, assault and aggravatedassault), tried marijuana before age 15, daily smoking before age 15, regular drinking before age 15 and any intercourse beforeage 15. It also includes measures of Self-Esteem and Locus of Control. Self-Esteem is measured using the ten-item Rosenbergscale administered in 1980. Locus of control is a measure of how much control an individual believes they have over their lifeand is measured using the 4-item Rotter scale. IQ and grades are from high school transcripts. IQ is pooled across several IQtests using IQ percentiles. GPA is the individual’s core-subject GPA from 9th grade. Outcomes: Due to the biennial nature ofthe survey after 1994, some respondents are not interviewed at age 35, for these individuals age 36 is used. Earnings includeszero-earners and excludes observations over $200,000 (2005 dollars). Hourly wage excludes observations less than $3 or over$200 (2005 dollars). Hours worked excludes observations less than 80 or more than 4000. Jail by age 35 indicates whether therespondent had listed residing in a jail or prison at some point before age 35. Welfare at age 35 indicates whether therespondent received any positive amount of welfare at age 35. Married at age 35 indicates whether the responded wascurrently married. BA degree by age 35 indicates whether the respondent received a BA degree (or higher) by age 35.Depression in 1992 is based on the 7-item Center for Epidemiologic Studies Depression Scale (CES-D). Sample: The sampleexcludes the military over sample. The samples differ across the IQ, AFQT, and GPA due to missing measures across thesamples.



Figure 3: Association of the Big Five and intelligence with years ofcompleted schooling

-0.1 -0.05 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35

Crystalized Intelligence

Fluid Intelligence

Openness

Conscientiousness

Extraversion

Agreeableness

Emotional Stability

Standardized Regression Coefficient

Males

Unadjusted for Intelligence Adjusted for Intelligence

Notes: The figure displays standardized regression coefficients from a multivariate regression of years of school attended onthe Big Five and intelligence, controlling for age and age squared. The bars represent standard errors. The Big Fivecoefficients are corrected for attenuation bias. The Big Five were measured in 2005. Years of schooling were measured in2008. Intelligence was measured in 2006. The measures of intelligence were based on components of the Wechsler AdultIntelligence Scale (WAIS). The data is a representative sample of German adults between the ages 21 and 94.Source: Almlund et al. (2011) German Socio-Economic Panel (GSOEP), waves 2004-2008.



IQ is highly predictive of performance on complex tasks andjobs (Gottfredson, 1997).

The importance of IQ increases with job complexity, defined asthe information processing requirements of the job: cognitiveskills are more important for professors, scientists, and seniormanagers than for semi-skilled or unskilled laborers (Schmidtand Hunter, 2004).

In contrast, the importance of Conscientiousness does not varymuch with job complexity (Barrick and Mount, 1991),suggesting that it pertains to a wider spectrum of jobs.



The Evolution of Personality Traits Over the Life Cycle



Figure 4: Cumulative Mean-Level Changes in Personality Across the LifeCycle

p1550 In contrast, a longitudinal study of adult intellectual development shows mean-leveldeclines in cognitive skills, particularly cognitive processing speed, after age 55 or so (Schaie,1994). Figure 1.23a shows mean-level changes in cognitive skills using a longitudinalanalysis, and Figure 1.23b shows mean-level changes using a cross-sectional analysis.210 As

Social vitality

−0.20

0.20.40.60.8

11.2

Age

Cum

ulat

ive

d va

lue

−0.20

0.20.40.60.8

11.2

Cum

ulat

ive

d va

lue

−0.20

0.20.40.60.8

11.2

Cum

ulat

ive

d va

lue

Agreeableness

Emotional stability

10 20 30 40 50 60 70 80

Age10 20 30 40 50 60 70 80

Age10 20 30 40 50 60 70 80

−0.20

0.20.40.60.8

11.2

Cum

ulat

ive

d va

lue

−0.20

0.20.40.60.8

11.2

Cum

ulat

ive

d va

lue

−0.20

0.20.40.60.8

11.2

Cum

ulat

ive

d va

lue

Social dominance

Conscientiousness

Openness to experience

Age10 20 30 40 50 60 70 80

Age10 20 30 40 50 60 70 80

Age10 20 30 40 50 60 70 80

f0115 Figure 1.22 Cumulative Mean-Level Changes in Personality across the Life Cycle.

Note: Social vitality and social dominance are aspects of Big Five Extraversion. Cumulative d valuesrepresent total lifetime change in units of standard deviations (“effect sizes”).Source: Figure taken from Roberts, Walton, and Viechtbauer (2006) and Roberts and Mroczek (2008).Reprinted with permission of the authors.

fn1055210 Cross-sectional estimates of mean-level change are biased by cohort effects (e.g., the Flynn effect), whereas longitu-

dinal estimates are biased by test–retest learning (when the same IQ tests are administered repeatedly to the same sub-jects) and by selective attrition. Thus, both estimates must be considered in conjunction as evidence for mean-levelchange.

Hanushek_2011 978-0-444-53444-6 00001

Personality Psychology and Economics 119

Note: Cumulative d values represent total lifetime change in units of standard deviations (“effect sizes”).Source: Figure taken from Roberts et al. (2006) and Roberts and Mroczek (2008). Reprinted with permission of the authors.



Causal Evidence

Problems with Establishing Causality

Most studies in personality psychology do not address thequestion of causality, i.e., do measured traits cause (rather thanjust predict) outcomes?Equation (1) shows how an outcome at age a, Ta, which is theperformance on a task, depends on cognition Ca, personalityPa, other acquired skills Ka, and the effort allocated to the taskeTa :

Ta︸︷︷︸Outcome on atask at age a

= φa( Ca︸︷︷︸Cognition

, Pa︸︷︷︸Personality

, Ka︸︷︷︸Other

Acquiredskills

, eTa︸︷︷︸Effort

devoted totask

) a = 1, . . . ,A.

(1)



Equation (2) shows how the effort allocated on the taskdepends on cognition Ca, personality Pa, other acquired skillsKa, incentives RTa , and preferences Υa.

Preferences can be thought of as additional traits:

eTa = ψTa(Ca,Pa,Ka, RTa︸︷︷︸Incentivesto perform

on task

, Υa︸︷︷︸Preferences

). (2)



Equations (1) and (2) formalize the threats to establishing acausal relationship between outcomes and traits.

Multiple traits, effort, and acquired skills generate performancein a given task.



The traits and acquired skills evolve over time throughinvestment and habituation.Equation (3) shows that traits at age a + 1 are a function ofcognitive ability, personality traits, other acquired skills, andinvestment Ia at age a.

(Ca+1,Pa+1,Ka+1) = ηa(Ca , Pa , Ka , Ia︸︷︷︸Investment

andexperience

), a = 1, . . . ,A.

(3)



Extreme Examples of Personality Change



Evidence from the GED Testing Program

The GED testing program currently produces 12% of highschool certificates each year in the United States.

Table 4 shows the correlations between GED scores and otherachievement test scores.



Table 4: Validities of GED TestTest Correlation Source(s)

Armed Forces Qualification Test (AFQT) 0.75 - 0.79 † Means and Laurence (1984)

Iowa Test of Educational Development 0.88 † Means and Laurence (1984)

American College Test (ACT) 0.80 † Means and Laurence (1984)

Adult Performance Level (APL) Survey 0.81 † Means and Laurence (1984)

New York’s Degrees of Reading Power (DRP) Test 0.77 † Means and Laurence (1984)

Test of Adult Basic Education (TABE) 0.66-0.68† Means and Laurence (1984)

General Aptitude Test Battery (GATB) 0.61-0.67† Means and Laurence (1984)

National Adult Literacy Survey (NALS) factor 0.78 ‡ Baldwin (1995)

† Uses mean GED subtest scores‡ Uses a general GED factor



GED recipients are smarter than other dropouts.

Figure 5 shows the distributions of a factor extracted from thecomponents of the Armed Services Vocational Aptitude Battery(ASVAB) for male high school dropouts, GED recipients, andhigh school graduates.



Figure 5: Cognitive ability by educational status

Source: Reproduced from Heckman et al. (2011), which uses data from the NationalLongitudinal Study of Youth 1979 (NLSY79). Notes: The distributions above representcognitive ability factors estimated using a subset of the Armed Services Vocational AptitudeBattery (ASVAB) and educational attainment as laid out in Hansen et al. (2004). The sampleis restricted to the cross-sectional subsample for both males and females. Distributions showonly those with no post-secondary educational attainment. The cognitive ability factors arenormalized by gender to be mean zero standard deviation one.



Figure 6 shows measures of early adolescent drug use, crime,sex, and violence extracted from three data sources.



Figure 6: Measures of Adolescent Behaviors for Male Dropouts, GEDRecipients, and High School Graduates: Smoking and Drinking

0.2

.4.6

Smokesby 15

(NLSY79)

Drinksby 15

(NLSY79)

Smokesby 14

(NLSY97)

Drinksby 14

(NLSY97)

SmokesGr.8

(NELS)

Binge DrinksGr.10

(NELS)

Drop GED HSG +/− S.E.

Sources: Heckman et al. (2012, Chapter 3). National Longitudinal Survey of Youth 1979, National Longitudinal Survey ofYouth 1997, National Educational Longitudinal Survey. school.Notes: Minor crime includes vandalism, shoplifting, pettytheft, fraud, holding or selling stolen goods. Major crime includes auto theft, breaking/entering private property, grand theft.Violent crime includes fighting, assault, aggravated assault.



Figure 6: Measures of Adolescent Behaviors for Male Dropouts, GEDRecipients, and High School Graduates: Sex and Violent Behavior

0.2

.4.6

Sexby 15

(NLSY79)

Fightby 14

(NLSY97)

Gangby 14

(NLSY97)

School FightGr.8

(NELS)





Figure 6: Measures of Adolescent Behaviors for Male Dropouts, GEDRecipients, and High School Graduates: Criminal Behavior

0.2

.4.6

.8

MinorCrime

(NLSY79)

MajorCrime

(NLSY79)

ViolentCrime

(NLSY79)

Arrestedby 14

(NLSY97)

Prop Crimeby 14

(NLSY97)

Theftby 14

(NLSY97)





Figure 7 summarizes these adolescent behaviors using a singlefactor and shows that unlike the cognitive summary measuresthe distribution of the noncognitive summary measure of GEDrecipients is much closer to that of dropouts.



Figure 7: Distribution of a Summary of Noncognitive Ability byEducation Group

Source: Reproduced from Heckman et al. (2011), which uses data from the National Longitudinal Study of Youth 1979(NLSY79). Notes: The distributions above represent non-cognitive ability factors estimated using measures of early violentcrime, minor crime, marijuana use, regular smoking, drinking, early sexual intercourse, and educational attainment as inHansen et al. (2004). Sample restricted to the cross-sectional subsample for both males and females. Distributions show onlythose with no post-secondary educational attainment. The non-cognitive ability factors normalized to be mean zero standarddeviation one.



Figure 8: Post-Secondary Educational Attainment Across EducationGroups Through Age 40 (NLSY79) - Males

0.2

.4.6

.8

Prop

ortio

n

SomeColl

Some Coll, MoreThan 1 Year

A.A. B.A.

GED HSG S.E.

Sources: Heckman et al. (2012, Chapter 4). National Longitudinal Survey of Youth 1979. Notes: The graph showspost-secondary educational attainment of GED recipients and high school graduates. Variable Definitions: “Some College”represents people who entered any post-secondary institution ever. “Some College, More Than a Year” represents people whocompleted at least a year of some post-secondary education ever. “A.A.” represents people who obtained associate’s degreesever. “B.A.” represents people who obtained bachelor’s degrees ever. “B.A.” also includes people with higher education:M.A. Ph.D and professional degrees. Tests of Significance: The estimates for GED recipients and high school graduates arestatistically significantly different at the 5% level for all but attainment of the A.A. degree.



Figure 9: Survival Rates in Various States for Male Dropouts, GEDRecipients, and High School Graduates: Survival Rate in Employment

.2

.4

.6

.8

1Su

rviv

al R

ate

in E

mpl

oyed

Sta

te

1 2 3 4 5 6 7 8 9 10

Years Since Start of Spell

Dropout GED HSG



Figure 9: Survival Rates in Various States for Male Dropouts, GEDRecipients, and High School Graduates: Survival Rate in Same Job

0

.2

.4

.6

.8

1Su

rviv

al R

ate

in C

urre

nt J

ob

1 2 3 4 5 6 7 8 9 10


Dropout GED HSG



Figure 9: Survival Rates in Various States for Male Dropouts, GEDRecipients, and High School Graduates: Survival Rate in Marriage

.4

.6

.8

1Su

rviv

al R

ate

in M

arri

age

1 2 3 4 5 6 7 8 9 10


Dropout GED HSG



Figure 9: Survival Rates in Various States for Male Dropouts, GEDRecipients, and High School Graduates: Survival Rate in Not HavingBeen Incarcerated

.85

.9

.95

1

Sur

viva

l Rat

e in

Non

−In

carc

erat

ed S

tate

1 2 3 4 5 6 7 8 9 10


Dropout GED HSG



Source: Heckman et al. (2012, Chapter 4). National Longitudinal Survey of Youth 1979(NLSY79), nationally representative cross sectional sample. Notes: The spell to first timebeing incarcerated begins in the first year that individuals exit school. Tests of Significance:The estimates for GED recipients and high school graduates are statistically significantlydifferent at the 5% level for all but the 2nd year of “Survival Rate in Not Having BeenIncarcerated.” The estimates for dropouts and high school graduates are statisticallysignificantly different at the 5% level for all but the 2nd year of “Survival Rate in Not HavingBeen Incarcerated.” The estimates for dropouts and GED are statistically only significantlydifferent at the 5% level for the 5th year of the “Survival Rate in Marriage.”



Figure 10 shows the hourly wages and annual earnings of maleGED recipients and high school graduates compared to highschool dropouts from age 20 to age 40.



Figure 10: Labor Market Outcomes Differences - By Age - NLSY79 -Males: Annual Earnings

010

000

2000

030

000

Ann

ual E

arni

ngs

Age 20 to 24 Age 25 to 29 Age 30 to 34 Age 35 to 39Raw Abil BG Raw Abil BG Raw Abil BG Raw Abil BG

GED HSG S.E.



Figure 10: Labor Market Outcomes Differences - By Age - NLSY79 -Males: Hourly Wage

−5

05

10

Hou

rly W

age

Age 20 to 24 Age 25 to 29 Age 30 to 34 Age 35 to 39Raw Abil BG Raw Abil BG Raw Abil BG Raw Abil BG

GED HSG S.E.



Source: Heckman et al. (2012, Chapter 3). National Longitudinal Survey of Youth 1979.Controls: “Raw” – age, race, and region of residence; “Abil” –age, race, region of residence,and AFQT adjusted for schooling at time of test; “BG” – mother’s highest grade completed,urban status at age 14, family income in 1978, broken home status at age 14, south at age 14,AFQT, and factors based on adolescent behavioral measures, crime and school performance.Regressions exclude those reporting earning more than $300,000 or working more than 4,000hours. Notes: All regressions allow for heteroskedastic errors and when appropriate clusteringat the individual level.



Evidence from The Perry Preschool Program and OtherInterventions

Participants were taught social skills in a “plan-do-review”sequence where students planned a task, executed it, and thenreviewed it with teachers and fellow students.

They learned to work with others when problems arose.



Figure 11: Perry Preschool Program: IQ, by Age and Treatment Group

traits of the participants were beneficially improved in a lasting way.11 This chapter isabout those traits.

p0025 Personality psychologists mainly focus on empirical associations between their mea-sures of personality traits and a variety of life outcomes. Yet for policy purposes, it isimportant to know mechanisms of causation to explore the viability of alternative poli-cies.12 We use economic theory to formalize the insights of personality psychology andto craft models that are useful for exploring the causal mechanisms that are needed forpolicy analysis.

p0030 We interpret personality as a strategy function for responding to life situations. Person-ality traits, along with other influences, produce measured personality as the output ofpersonality strategy functions. We discuss how psychologists use measurements of theperformance of persons on tasks or in taking actions to identify personality traits andcognitive traits. We discuss fundamental identification problems that arise in applyingtheir procedures to infer traits.

p0035 Many economists, especially behavioral economists, are not convinced about thepredictive validity, stability, or causal status of economic preference parameters or per-sonality traits. They believe, instead, that the constraints and incentives in situations

100

IQ

95

90

85

80

75

4Entry

78.5

83.3 83.5

86.3 87.1 86.9 86.884.6

8587.788.1

91.791.3

94.995.5

5 6

Age

Treatment group

Control group

7 8 9 10

79.6

f0010 Figure 1.1 Perry Preschool Program: IQ, by Age and Treatment Group.

Notes: IQ measured on the Stanford–Binet Intelligence Scale (Terman and Merrill, 1960). The test wasadministered at program entry and at each of the ages indicated.Source: Cunha, Heckman, Lochner, and Masterov (2006) and Heckman and Masterov (2007) based ondata provided by the High Scope Foundation.

fn006011 We discuss this evidence in Section 8. The traits changed were related to self-control and social behavior. Participants

of both genders had better “externalizing behavior,” while for girls there was also improvement in Openness toExperience. See Heckman, Malofeeva, Pinto, and Savelyev (first draft 2008, revised 2011). Duncan and Magnuson(2010) offer a different interpretation of the traits changed by the Perry experiment. But both analyses agree that itwas not a boost in IQ that improved the life outcomes of Perry treatment group members.

fn006512 See Heckman (2008a).

Hanushek_2011 978-0-444-53444-6 00001

Personality Psychology and Economics 5

Notes: IQ measured on the Stanford-Binet Intelligence Scale (Terman and Merrill, 1960). The test was administered atprogram entry and at each of the ages indicated.Source: Cunha et al. (2006) and Heckman and Masterov (2007) based on data provided by the High Scope Foundation.



The Perry Preschool Program worked primarily throughimproving personality traits.

Participants had better direct measures of personal behavior (aweighted average of “absences and truancies,” “lying andcheating,” “stealing,” and “swears or uses obscene words”measured by teachers in the elementary school years).



Figure 12: Decompositions of Treatment Effects on Outcomes, Males

.136

.062

.071

.071 .557

.161

.088

.144

.246

.114

.013

# f i d 40 ( )

# of adult arrests (misd.+fel.) , age 27 (‐)

# of felony arrests, age 27 (‐)

# of misdemeanor arrests, age 27 (‐)

CAT total⁽¹⁾, age 14 (+)

.077

.086

.056

.204

.149

.403

0% 20% 40% 60% 80% 100%

# of lifetime arrests, age 40 (‐)

# of adult arrests (misd.+fel.), age 40 (‐)


# of misdemeanor arrests, age 40 (‐)

Cognitive Factor Externalizing Behavior Academic Motivation Other Factors

Notes: The total treatment effect is normalized to 100%. One-sided p-values are shown above each component in eachoutcome. “(+)” and “(-)” denote positive and negative total treatment effects. “CAT total” denotes California AchievementTest total score.



Figure 13: Decompositions of Treatment Effects on Outcomes, Females

.256

.152

.120

.099

.527

.056

.319

.306

.231

.282

# of felony arrests , age 27 (‐)

# of misd. violent crimes cited at arrest, age 27 (‐)

CAT total, age 14 (+)

CAT total, age 8 (+)

.046

.050

.066

.320

.369

.372

0% 20% 40% 60% 80% 100%

# of lifetime violent crimes cited at arrest, age 40 (‐)


# of misd. violent crimes cited at arrest, age 40 (‐)

Cognitive Factor Externalizing Behavior Academic Motivation Other Factors

Notes: The total treatment effect is normalized to 100%. One-sided p-values are shown above each component in eachoutcome. “(+)” and “(-)” denote positive and negative total treatment effects. “CAT total” denotes California AchievementTest total score.



Figure 14: Decompositions of Treatment Effects, Factor Scores versusMLE

.091

.071

.025

.050

.020

.066

.048

.120

.020

.099

.342

.557

.072

.114

.230

.369

.196

.372

.231

.319

.129

.306

SCORE

MLE

SCORE

MLE

SCORE

MLE

SCORE

MLE⁽²⁾

SCORE⁽¹⁾

Females

# of misdemeanor

violent crimes, age 27 (‐)

# of felony arrests,

age 27 (‐)

# of misdemeanor

violent crimes, age 40 (‐)


age 40 (‐)

# of misdemeanor

.079

.056

.132

.136

.085

.071

.091

.475

.342

.378

.403

.067

.088

.223

.246

.072

0% 20% 40% 60% 80% 100%

MLE

SCORE

MLE

SCORE

MLE

SCORE

MLE

Cognition Externalizing Behavior Academic Motivation Other Factors

Males

arrests, age 27 (‐)


age 40 (‐)


age 27 (‐)

# of misdemeanor

arrests, age 40 (‐)



Summary

Success in life depends on a multiplicity of traits, not just thosemeasured by IQ, grades, and standardized achievements tests.

All psychological traits are measured through behavior orperformance on a task.

Performance on tasks depends on incentives and multiple traits,giving rise to a fundamental identification problem whenmeasuring any one trait.



Different tasks require different bundles of cognitive andpersonality traits.

Given their endowments of traits and the incentives, people sortinto tasks in life in pursuit of their comparative advantage.

Traits are stable across situations, but their manifestationdepends on the incentives in any situation where they aremeasured as well as on other traits and skills.

Scores on achievement tests capture both cognitive andpersonality traits.



The evidence in this paper should give pause to analysts andpolicy makers who rely solely on achievement tests to monitorschool performance and school systems.

Standardized achievement tests do not fully capture other skillsthat matter in life.

GED recipients perform about as well as high school graduateson achievement tests but perform much worse in many aspectsof life because they lack important personality traits.

Categorizing GED recipients as high school graduatesmisrepresents national statistics on educational attainment.



The bundle of traits captured by scores on achievement testsdoes not accurately measure the diverse traits required forsuccess in life.Interventions that promote beneficial changes in personalityhave an important place in a portfolio of public policies tofoster human development.

ACT, Inc. (2007). The ACT Technical Manual. Iowa City, IA: ACT,Inc.

Almlund, M., A. Duckworth, J. J. Heckman, and T. Kautz (2011).Personality psychology and economics. In E. A. Hanushek,S. Machin, and L. Woßmann (Eds.), Handbook of the Economicsof Education, Volume 4, pp. 1–181. Amsterdam: Elsevier.

Baldwin, J. (1995). Who Took the GED? GED 1994 StatisticalReport. Washington, D.C.: American Council on Education, GEDTesting Service, Center for Adult Learning.

Barrick, M. R. and M. K. Mount (1991). The Big Five personalitydimensions and job performance: A meta-analysis. PersonnelPsychology 44(1), 1–26.

Borghans, L., B. H. H. Golsteyn, J. J. Heckman, and J. E.Humphries (2011). Identification problems in personalitypsychology. Personality and Individual Differences 51(SpecialIssue on Personality and Economics), 315–320. E. Ferguson, J.J.Heckman, and P. Corr, editors.

Costa, P. T. and R. R. McCrae (1992). Revised NEO PersonalityInventory (NEO PI-R) and the NEO Five-Factor Inventory(NEO-FFI) professional manual. Odessa, FL: PsychologicalAssessment Resources.

Cunha, F., J. J. Heckman, L. J. Lochner, and D. V. Masterov(2006). Interpreting the evidence on life cycle skill formation. InE. A. Hanushek and F. Welch (Eds.), Handbook of the Economicsof Education, Chapter 12, pp. 697–812. Amsterdam:North-Holland.

Feingold, A. (1982). The validity of the information and vocabularysubtests of the WAIS. Journal of Clinical Psychology 38(1),169–174.

GED Testing Service (2009, March). The technical manual: 2002series GED tests. Technical manual, American Council onEducation and GED Testing Service, Washington, DC.http://www.acenet.edu/Content/NavigationMenu/ged/

pubs/TechnicalManual_2002SeriesGEDTests.pdf.Gottfredson, L. S. (1997, January-February). Why g matters: The

complexity of everyday life. Intelligence 24(1), 79–132.Gough, H. G. and A. B. Heilbrun (1983). The Adjective Check List

Manual. Palo Alto, CA: Consulting Psychologists Press.Greene, A. C., G. L. Sapp, and B. Chissom (1990). Validation of the

Stanford-Binet Intelligence Scale: Fourth edition with exceptionalblack male students. Psychology in the Schools 27(1), 35–41.

Hansen, K. T., J. J. Heckman, and K. J. Mullen (2004,July–August). The effect of schooling and ability on achievementtest scores. Journal of Econometrics 121(1-2), 39–98.

Hartlage, L. C. and C. T. Steele (1977). WISC and WISC-Rcorrelates of academic achievement. Psychology in theSchools 14(1), 15–18.

Heckman, J. J., J. E. Humphries, and T. Kautz (2012). The GEDand the Problem of Soft Skills. Unpublished book manuscript,University of Chicago, Department of Economics.

Heckman, J. J., J. E. Humphries, S. Urzua, and G. Veramendi(2011). The effects of educational choices on labor market,health, and social outcomes. Unpublished manuscript, Universityof Chicago, Department of Economics.

Heckman, J. J. and D. V. Masterov (2007). The productivityargument for investing in young children. Review of AgriculturalEconomics 29(3), 446–493.

John, O. P. and S. Srivastava (1999). The big five trait taxonomy:History, measurement and theoretical perspectives. In L. A. Pervinand O. P. John (Eds.), Handbook of Personality: Theory andResearch, Chapter 4, pp. 102–138. New York: The Guilford Press.

Kobrin, J. L., B. F. Patterson, E. J. Shaw, K. D. Mattern, andS. M. Barbuti (2008). Validity of the SAT for predicting first-yearcollege grade point average. The College Board 5, 1–10.

Means, B. and J. H. Laurence (1984). Characteristics andperformance of recruits enlisted with general educationaldevelopment (ged) credentials. Technical Report FR-PRD-84-6,Human Resources Research Organization, Alexandria, VA.

Michalko, K. T. and D. H. Saklofske (1996). A psychometricinvestigation of the Wechsler Individual Achievement Test with asample of Saskatchewan schoolchildren. Canadian Journal ofSchool Psychology 12(1), 44–54.

O’Leary, U.-M., K. M. Rusch, and S. J. Guastello (1991).Estimating age-stratified WAIS-R IQs from scores on the Raven’sStandard Progressive Matrices. Journal of ClinicalPsychology 47(2), 277–284.

Omizo, M. M. (1980). The differential aptitude tests as predictorsof success in a high school for engineering program. Educationaland Psychological Measurement 40(1), 197–203.

Roberts, B. W. and D. Mroczek (2008). Personality trait change inadulthood. Current Directions in Psychological Science 17(1),31–35.

Roberts, B. W., K. E. Walton, and W. Viechtbauer (2006). Patternsof mean-level change in personality traits across the life course: Ameta-analysis of longitudinal studies. PsychologicalBulletin 132(1), 1–25.

Rothlisberg, B. (1987). Comparing the Stanford-Binet, fourthedition to the WISC-R: A concurrent validity study. Journal ofSchool Psychology 25(2), 193–196.

Schmidt, F. L. and J. Hunter (2004). General mental ability in theworld of work: Occupational attainment and job performance.Journal of Personality and Social Psychology 86(1), 162–173.

Terman, L. M. and M. A. Merrill (1960). Stanford-Binet IntelligenceScale: Manual for the Third Revision Form L-M. Boston:Houghton Mifflin.


http://www.acenet.edu/Content/NavigationMenu/ged/pubs/TechnicalManual_2002SeriesGEDTests.pdf

http://www.acenet.edu/Content/NavigationMenu/ged/pubs/TechnicalManual_2002SeriesGEDTests.pdf

Date post:	14-Aug-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Hard Evidence on Soft Skills - University of Chicago · 2012-03-29 · Hard Evidence on Soft Skills...

Documents