What Works? A Meta Analysis of Recent Active Labor Market Program Evaluations · 2017-04-05 ·...

NBER WORKING PAPER SERIES

WHAT WORKS? A META ANALYSIS OF RECENT ACTIVE LABOR MARKET PROGRAM EVALUATIONS

David CardJochen KluveAndrea Weber

Working Paper 21431http://www.nber.org/papers/w21431

NATIONAL BUREAU OF ECONOMIC RESEARCH1050 Massachusetts Avenue

Cambridge, MA 02138July 2015, Revised April 2017

We are extremely grateful to the editor and five referees for helpful comments on an earlier draft, and to seminar participants at IRVAPP Trento, ILO Geneva, OECD Paris, European Commission Brussels, The World Bank Washington DC, University of Oslo, ISF Oslo, MAER-Net 2015 Prague Colloquium, IFAU Uppsala. We also thank Diana Beyer, Hannah Frings and Jonas Jessen for excellent research assistance. Financial support from the Fritz Thyssen Foundation and the Leibniz Association is gratefully acknowledged. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.

NBER working papers are circulated for discussion and comment purposes. They have not been peer-reviewed or been subject to the review by the NBER Board of Directors that accompanies official NBER publications.

© 2015 by David Card, Jochen Kluve, and Andrea Weber. All rights reserved. Short sections of text, not to exceed two paragraphs, may be quoted without explicit permission provided that full credit, including © notice, is given to the source.

What Works? A Meta Analysis of Recent Active Labor Market Program Evaluations David Card, Jochen Kluve, and Andrea WeberNBER Working Paper No. 21431July 2015, Revised April 2017JEL No. J08,J24

ABSTRACT

We summarize the estimates from over 200 recent studies of active labor market programs. We classify the estimates by type of program and participant group, and distinguish between three different post-program time horizons. Using regression models for the estimated program effect (for studies that model the probability of employment) and for the sign and significance of the estimated effect (for all the studies in our sample) we conclude that: (1) average impacts are close to zero in the short run, but become more positive 2-3 years after completion of the program; (2) the time profile of impacts varies by type of program, with larger average gains for programs that emphasize human capital accumulation; (3) there is systematic heterogeneity across participant groups, with larger impacts for females and participants who enter from long term unemployment; (4) active labor market programs are more likely to show positive impacts in a recession.

David CardDepartment of Economics549 Evans Hall, #3880University of California, BerkeleyBerkeley, CA 94720-3880and [email protected]

Jochen KluveHumboldt-University and RWI Spandauer Str. 1 10178 Berlin Germany [email protected]

Andrea WeberVienna University of Economics and BusinessEconomics DepartmentWelthandelsplatz 11020 [email protected]

1

InthelongperiodofrecoveryaftertheGreatRecessionthereisrenewed

interestinthepotentialuseofactivelabormarketpolicies(ALMPs)tohelpeaseawide

rangeoflabormarketproblems,includingyouthunemploymentandpersistent

joblessnessamongdisplacedadults(e.g.,Martin,2014).Althoughtrainingprograms,

employmentsubsidies,andsimilarpolicieshavebeeninuseforwellover50years,

credibleevidenceontheircausalimpactshasonlybecomeavailableinrecentdecades

(seeLalonde2003forabriefhistory).Withinarelativelyshortperiodoftimethe

numberofscientificevaluationshasexploded,offeringthepotentialtolearnwhattypes

ofprogramsworkbest,inwhatcircumstances,andforwhom.

InthispaperwesynthesizetherecentALMPevaluationliterature,lookingfor

systematicevidenceontheseissues.1Weextendthesampleusedinourearlieranalysis

(Card,Kluve,Weber,2010;hereafterCKW),doublingthenumberofstudies(from97to

207)andincreasingthenumberofseparateprogramestimatesfrom343to857.Many

ofthelatestALMPstudiesmeasureimpactsontheemploymentrateofparticipants,

yieldingover350estimatesforthisoutcomethatcanbereadilycomparedacross

studies.

Thisnewsampleofestimatesallowsusextendourearlierworkin4mainways.

First,wecanmorepreciselycharacterizeaverageprogramimpactsbytypeofALMPand

post-programtimehorizon.Second,weareabletocomparetherelativeefficacyof

differenttypesofALMP’s(e.g.trainingversusjobsearchassistance)fordifferent

participantgroups(e.g.,youthsversusolderworkers).Third,weprovidenewevidence

onthevariationinprogrameffectsatdifferentpointsinthebusinesscycle.Finally,we

1PreviousreviewsincludeHeckman,LalondeandSmith(1999),whosummarize75microeconometricevaluationsfromtheU.S.andothercountries,Kluve(2010),whoreviewscloseto100studiesfromEurope,andFilgesetal.(2015),whoanalyzeanarrowersetof39studies.Greenberg,MichalopoulosandRobins(2003)reviewU.S.programstargetedtodisadvantagedworkers.BergemannandvandenBerg(2008)surveyprogrameffectsbygender.IbarraránandRosas(2009)reviewprogramsinLatinAmericasupportedbytheInter-AmericanDevelopmentBank.RelatedmetaanalysesfocusingonlabormarketinterventionsinlowandmiddleincomecountriesincludeChoandHonorati(2014)andGrimmandPaffhausen(2015).

2

conductasystematicanalysisofpotentialpublicationbiasesintherecentALMP

literature.

Wesummarizetheestimatesfromdifferentstudiesintwocomplementaryways.

Ourmainapproachistoexaminetheestimatedprogrameffectsonemployment,

ignoringthefindingsfromstudiesthatmodelotheroutcomes(suchasthedurationof

timetoanunsubsidizedjob).Oursecondapproach--whichcanbeappliedtoallthe

estimatesinoursample,regardlessoftheoutcomevariable--istoclassify"signand

significance"basedonwhethertheestimatedimpactissignificantlypositive,statistically

insignificant,orsignificantlynegative.Thenarrowerfocusofthefirstapproachis

preferredinthemetaanalysisliterature(e.g.,HedgesandOlkin,1985;Robertsand

Stanley,2005;StanleyandDoucouliagos,2012),becausethemagnitudeoftheeffectis

notmechanicallyrelatedtothenumberofobservationsusedinthestudy,whereas

statisticalsignificanceis(inprinciple)sample-sizedependent.Fortunately,thetwo

approachesyieldsimilarconclusionswhenappliedtothesubsetofstudiesforwhich

employmenteffectsareavailable,givingusconfidencethatourmainfindingsare

invarianttohowwesummarizetheliterature.

Wereachfourmainsubstantiveconclusions.First,consistentwiththepattern

documentedinCKW,wefindthatALMPshaverelativelysmallaverageeffectsinthe

shortrun(lessthanayearaftertheendoftheprogram),butlargeraverageeffectsin

themediumrun(1-2yearspostprogram)andlongerrun(2+years).Acrossstudiesthat

modelimpactsonemployment,theshortrunimpactsarecenteredbetween1and3

percentagepoints(ppt.)Thedistributionofmediumruneffectsisshiftedtotheright,

centeredaround3to5ppt.,whilethelongerruneffectsarecenteredbetween5and12

ppt.Asabenchmark,thegapinemploymentratesbetweenU.S.menwithonlyahigh

schooleducationandthosewitha2or3yearcommunitycollegedegreeis10ppts.,

suggestingthata5-10ppt.longer-runimpactiseconomicallymeaningful.2

2In2015,theaveragemonthlyemploymentrateofmenoverage25withahighschooleducationintheU.S.was63.5%;theaverageformenwithanAssociatedegreewas73.8%.(U.S.DOL,2016).

3

Second,thetimeprofileofaverageimpactsinthepost-programperiodvaries

withthetypeofALMP.Jobsearchassistanceprogramsthatemphasize"workfirst"

tendtohavesimilarimpactsintheshortandlongrun,whereastrainingandprivate

sectoremploymentprogramshavelargeraverageeffectsinthemediumandlonger

runs.Publicsectoremploymentsubsidiestendtohavesmallorevennegativeaverage

impactsatallhorizons.

Third,wefindthattheaverageimpactsofALMPsvaryacrossgroups,withlarger

averageeffectsforfemalesandparticipantsdrawnfromthepooloflongterm

unemployed,andsmalleraverageeffectsforolderworkersandyouths.Wealsofind

suggestiveevidencethatcertainprogramsworkbetterforspecificsubgroupsof

participants.Jobsearchassistanceprogramsappeartoberelativelymoresuccessfulfor

disadvantagedparticipants,whereastrainingandprivatesectoremploymentsubsidies

tendtohavelargeraverageeffectsforthelongtermunemployed.Finally,comparing

therelativeefficacyofALMPsofferedatdifferentpointsinthebusinesscycle,wefind

thatprogramsinrecessionaryperiodstendtohavelargeraverageimpacts,particularly

ifthedownturnisrelativelyshort-lived.

Onthemethodologicalside,wefindthattheaverageprogrameffectsfrom

randomizedexperimentsarenotverydifferentfromtheaverageeffectsfromnon-

experimentaldesigns.Thisisreassuringgivenlongstandingconcernsoverthereliability

ofnon-experimentalmethodsforevaluatingjobtrainingandrelatedprograms(e.g.,

Ashenfelter,1987).Wealsofindthatthereissubstantialunobservedheterogeneityin

theestimatedprogramimpactsintheliterature.Thisheterogeneityislargerelativeto

thevariationattributabletosamplingerror,leadingtorelativelywidedispersioninthe

estimatedimpactsfromdesignswithsimilarprecision.Incontrasttothepatterns

uncoveredinmetaanalysesofminimumwageeffects(DoucouliagosandStanley,2009)

andtheintertemporalsubstitutionelasticity(Havranek,2015)thisdispersionisalso

nearlysymmetric.Asaresult,standardtestsforpublicationbias,whichlookfor

asymmetryinthedistributionofprogramestimates,areinsignificant.

4

II.SampleConstruction

a.SamplingImpactEvaluationStudies

WeextendthesampleinCKW,usingthesamecriteriatoselectin-scopestudies

andthesameprotocolstoextractinformationaboutprogramfeaturesandimpacts.The

CKWsamplewasderivedfromresponsestoa2007surveyofresearchersaffiliatedwith

theInstitutefortheStudyofLabor(IZA)andtheNationalBureauofEconomicResearch

(NBER)askingaboutevaluationstudieswrittenafter1995.3Toextendthissamplewe

beganbyreviewingtheresearchprofilesandhomepagesofIZAresearchfellowswitha

declaredinterestin“programevaluation”,lookingforstudieswrittensince2007.We

alsosearchedtheNBERworkingpaperdatabaseusingthesearchstrings“training”,

“active”,“publicsectoremployment”,and“searchassistance.”

InasecondstepweusedaGoogleScholarsearchtoidentifyallpapersciting

CKWortheearlierreviewbyKluve(2010).WealsosearchedthroughtheInternational

InitiativeforImpactEvaluation's"RepositoryofImpactEvaluationPublishedStudies,"

theonlineprojectlistoftheAbdulLatifJameelPovertyActionLab(J-PAL),andthelistof

LatinAmericanprogramevaluationsreviewedbyIbarraránandRosas(2009).

Afteridentifyinganinitialsampleofstudies,wereviewedthecitationsinallthe

paperstofindanyadditionalALMPstudies.Wealsoidentifiedfouradditionalpapers

presentedataconferenceinearlyfall2014.ThesearchprocesslastedfromAprilto

October2014andyielded154newstudiesthatwereconsideredforinclusioninour

ALMPimpactevaluationdatabase.

b.InclusionCriteria

3The1995startingpointwasdeterminedinpartbytheexistenceofseveralwell-knownsummariesoftheliteratureuptothemid-1990s,includingFriedlander,GreenbergandRobins(1997),Heckman,LalondeandSmith(1999),andGreenberg,MichalopoulosandRobins(2003).

5

Inordertogenerateaconsistentdatabaseacrossthetwowavesofdata

collection(2007and2014)weimposedthesamerestrictionsadoptedinCKW.First,the

program(s)analyzedintheevaluationhadtobeoneoffollowingfivetypes:

• classroomoron-the-jobtraining

• jobsearchassistance,monitoring,orsanctionsforfailingtosearch

• subsidizedprivatesectoremployment

• subsidizedpublicsectoremployment

• otherprogramscombiningtwoormoreoftheabovetypes.4

Sinceourfocusison"active"labormarketpolicies,weexcludestudiesoffinancial

incentives,suchasre-employmentbonuses(summarizedinMeyer,1995)orearnings

subsidyprograms(discussedinBlank,CardandRobins2000).Wealsoexcludeopen-

endedentitlementprogramslikechildcaresubsidies,andincludeonlyindividually

targetedemployersubsidyprograms,excludingtaxincentivesorothersubsidiesthat

areavailableforallnewlyhiredorexistingworkers.Finally,weexcludestudiesthat

reviseorupdateanolderstudyintheCKWsample,orhavesubstantialoverlapwithan

olderstudy.Methodologically,weincludeonlywell-documentedstudiesthatuse

individualmicrodataandincorporateacounterfactual/controlgroupdesignorsome

formofselectioncorrection.

Imposingthesecriteriaweretain110ofthe154studiesidentifiedinthesearch

process.5Weaddedthesetothe97studiesfromCKW,yieldingafinalsampleof207

impactevaluations.AcompletelistofthesestudiesiscontainedintheonlineData

Appendix,alongwithourentiredatabaseofprogramestimates.

Weemphasizethattheevaluationsinoursamplehavemanylimitations.At

best,thesestudiesmeasurethepartialequilibriumeffectsofALMPs,comparingthe

meanoutcomesinatreatmentgrouptothoseofanuntreatedcontrolorcomparison4Mostoftheseprogramscombineanelementofjobsearchwithtrainingorsubsidizedemployment.Wealsoinclude7estimatesofthe“threatofassignment“toaprograminthiscategory.5Themainreasonsforexclusionwere:overlapwithotherpapers(i.e.estimatingimpactsforthesameprogram);programoutofscope;andnoexplicitcounterfactualdesign.

6

group.6Evenfromthisnarrowperspectivefewstudiespresentinformationonthecosts

oftheprogram,anddetailedcost-benefitcalculationsareveryrare.Moreover,although

werestrictattentiontostudieswithacomparisongrouporselectioncorrectiondesign,

wesuspectthattheremaybesomebiasintheestimatesfromanyparticularstudy.We

donotbelieve,however,thatauthorshaveastrongincentivetochoosespecifications

thatleadtopositiveprogramestimates,sincemanywell-knownstudiesintheliterature

reportinsignificantorevennegativeimpactsforsomeprogramsorsubgroups(e.g.,

Bloometal.,2007).Thus,wedonothaveastrongpresumptionthatthebiasesinthe

literaturetendtobeone-sided.

c.ExtractingImpactEstimatesandInformationonProgramsandParticipants

Thenextstepwastoextractinformationabouttheprogramsandparticipants

analyzedineachstudy,andthecorrespondingprogramimpactestimates.7Usingthe

classificationsystemdevelopedinCKW,wegatheredinformationonthetypeofALMP,

onthetypesofparticipantsthatareadmittedtotheprogram(longtermunemployed,

regularunemploymentinsurancerecipients,ordisadvantagedindividuals8),thetypeof

dependentvariableusedtomeasuretheimpactoftheprogram,andtheeconometric

methodology.Wealsogatheredinformationonthe(approximate)datesofoperationof

theprogram,theageandgenderofparticipantsintheprogram,thesourceofthedata

usedintheevaluation(administrativerecordsoraspecializedsurvey),andthe

approximatedurationoftheprogram.

Ifastudyreportedseparateimpactestimateseitherbyprogramtypeorby

participantgroup,weidentifiedtheprogram/participantsubgroup(PPS)andcodedthe

impactestimatesseparately.Overall,wehaveinformationon526separatePPSsfrom6TheliteratureontheequilibriumeffectsofALMPisscarce.Foranotableexception,seeCreponetal.(2013).7AsinCKW,weextractedtheinformationfromthestudiesourselves,sincewefoundthatsubstantialknowledgeofevaluationmethodologyandtheALMPliteratureisoftenneededtointerpretthestudies.8Weclassifytheintakegroupas"disadvantaged"ifparticipantsareselectedfromlow-incomeorlow-labormarketattachmentindividuals.

7

the207studies,withaminimumof1andamaximumof10PPSsineachstudy.Wealso

identifieduptothreeimpactestimatesforeachPPS,correspondingtothreedifferent

post-programtimehorizons:short-term(approximatelyoneyearaftercompletionof

theprogram);mediumterm(approximatelytwoyearsafter);andlonger-term

(approximately3yearsafter).Intotal,wehave857separateprogramestimatesforthe

526program/participantsubgroups,withbetweenoneandthreeestimatesoftheeffect

oftheprogramatdifferenttimehorizons.9

Weusetwocomplementaryapproachestoquantifytheestimatedprogram

impacts.First,weclassifytheestimatesassignificantlypositive,insignificantlydifferent

fromzero,orsignificantlynegative(atthe5%level).Thismeasureofeffectivenessis

availableforeveryestimateinourdatabase.Forthesubsetofstudiesthatmeasure

effectsontheprobabilityofemployment,wealsoextractanestimateoftheprogram

effectontheemploymentrateofparticipants.10

Thefinalstepinourdataassemblyprocedurewastoaddinformationonlabor

marketconditionsatthetimeofoperationoftheprogram.Specifically,wegathered

informationonGDPgrowthratesandunemploymentratesfromtheOECD,theWorld

Bank,andtheILO.Forourmainanalysiswefocusonhowprogrameffectivenessis

relatedtotheaveragegrowthrateandtheaverageunemploymentrateduringthe

periodtheprogramgroupparticipatedintheALMP,thoughwealsolookattheeffectof

conditionsinthepost-programperiod.

9ForaspecificPPSandtimehorizonwetrytoidentifyandcodethemainestimateinthestudy.WedonotincludemultipleestimatesforthesamePPSandtimehorizon.10Wealsoextracttheaverageemploymentrateofthecomparisongroup,andforsomeanalysiswemodeltheprogrameffectdividedbyeitherthecomparisongroupemploymentrateorthestandarddeviationofthecomparisongroupemploymentrate.

8

III.DescriptiveOverview

a.ProgramTypes,ParticipantCharacteristics,EvaluationDesign

Table1presentsanoverviewoftheprogramestimatesinourfinalsample.As

noted,wehaveatotalof857differentimpactestimatesfor526differentPPSs

(program-type/participantsubgroupcombinations)extractedfrom207separate

studies.Todealwithpotentialcorrelationsbetweentheprogramestimatesfroma

givenstudy--arisingforexamplefromidiosyncraticfeaturesoftheevaluation

methodology--wecalculatestandarderrorsclusteringbystudy.

Column1presentsthecharacteristicsofouroverallsample,whilecolumns2-6

summarizetheestimatesfromfivecountrygroups:theGermaniccountries(Austria,

GermanyandSwitzerland),whichaccountforaboutonequarterofallstudies;the

Nordiccountries(Denmark,Finland,NorwayandSweden),whichaccountforanother

quarterofstudies;theAnglocountries(Australia,Canada,NewZealand,U.K.andU.S.),

whichaccountforjustover10%ofstudies;andtwonon-mutuallyexclusivegroupsof

lower/middleincomecountries--"non-OECD"countries(10%ofstudies),andLatin

AmericanandCaribbean(LAC)countries(10%ofstudies).AppendixFigure1showsthe

numbersofestimatesbycountry.ThelargestsourcecountriesareGermany(253

estimates),Denmark(115estimates),Sweden(66estimates),theU.S.(57estimates)

andFrance(42estimates).

ThesecondpanelofTable1showsthedistributionofprogramtypesinour

sample.Trainingprograms(includingclassroomandon-the-jobtraining)accountfor

aboutonehalfoftheprogramestimates,withbiggersharesinthenon-OECDandLAC

countries.Publicsectoremploymentprograms,bycomparison,arerelativelyrare

amongrecentevaluations,whilejobsearchassistance(JSA)programs,private

employmentsubsidiesandother/combinedprogramseachrepresentabout15%ofthe

estimates.11

11TheJSAcategoryincludesasmallnumberofevaluations(withatotalof8programestimates)forprogramsthatmonitorsearchactivityandthreatensanctionsforlowsearcheffort.Wecombinethese

9

Thenextthreepanelsofthetableshowthecharacteristicsoftheprogram

participants,classifiedbyagegroup,gender,and"type"ofparticipant.Aboutone-half

oftheestimatesareformixedageandmixedgendergroups,butwealsohaverelatively

largesubsetsofestimatesthatarespecifictoeitheryoungerorolderworkers,or

femalesormales.Sixty-fivepercentoftheprogramestimates(andnearlyallthe

estimatesfromtheGermaniccountries)areforparticipantswhoenterfromthe

unemploymentinsurance(UI)system.Typicallytheseparticipantsareassignedtoa

programandrequiredtoattendasaconditionforcontinuingbenefiteligibility.12The

remaining35%ofestimatesaresplitbetweenprogramsthatservethelongterm

unemployed(LTU)andthosethatservedisadvantagedparticipantgroups.Inmany

cases,thesegroupsarerecruitedbyprogramoperatorsandenrollvoluntarily.Such

voluntaryprogramsaremorecommonintheAngloSaxoncountriesandinless

developedcountriesthatlackaformalUIsystem.13

Nextweshowtheoutcomevariablesusedtomeasuretheprogramimpactand

thetimehorizonsoftheestimate.Themostcommonoutcome–particularlyinthe

Germanicandnon-OECDcountries–istheprobabilityofemployment,whilethelevelof

earningsisthemostcommonmetricintheAngloSaxoncountries.Aboutonesixthof

theprogramestimates–but40%ofthosefromNordiccountries–measuretheexitrate

fromthebenefitsystem,typicallyfocusingontherateofexittoanew(unsubsidized)

job.Finally,asmallsubsetofestimates–mostlyfromAngloSaxoncountries–focuson

theprobabilityofunemployment.Aboutonehalfoftheestimatesareforashortterm

horizon(<1year)afterprogramcompletion,35%foramediumterm(1-2years),and

16%foralongerterm(morethan2yearafter).

withJSAprogramsbecausebothtypesofprogramshavesimilarincentiveeffectsonparticipants'searchactivity.12ThistypeofprogramrequirementiswidespreadinEurope--seeSianesi(2004)foradiscussion.13TheU.S.jobtrainingprogramsanalyzedintheseminalpapersofAshenfelter(1978),AshenfelterandCard(1985),Lalonde(1986),Heckman,Ichimura,Smith,andTodd(1998)areallofthistype.

10

ThelastrowoftheTableshowsthefractionofprogramestimatesthatarebased

onanexperimentaldesign.Inmostofourcountrygroupsabout30%ofestimatescome

fromrandomizedcontrolledtrials(RCTs)thathavebeenexplicitlydesignedtomeasure

theeffectivenessoftheALMPofinterest.AnimportantexceptionistheGermanic

countries,wherenoexperimentallybasedestimatesareyetavailable.

Thedistributionofprogramestimatesovertime(definingtimebytheearliest

intakeyearoftheprogram)isshowninFigure1,withseparatecountsforexperimental

andnon-experimentalestimates.Oursampleincludesprogramsfromasfarbackas

1980,thoughthemajorityofestimatesarefromthe1990sandearly2000s,reflecting

ourfocusonstudieswrittensince1995.Thereisclearevidenceofatrendtoward

increasinguseofexperimentaldesigns:amongthe210estimatesfrom2004andlater,

61%arefromrandomizeddesigns.

b.MeasuresofProgramImpact-Overview

Table2givesanoverviewofourtwomainmeasuresofprogramimpact,

contrastingresultsfortheshortterm,mediumterm,andlongterm.Columnone

summarizesthesignandsignificanceofalltheavailableprogramestimates.Amongthe

415shorttermestimates,40%aresignificantlypositive,42%areinsignificant,and18%

aresignificantlynegative.Thepatternofresultsismorepositiveinthemediumand

longerterms,withamajorityofestimates(52%)beingsignificantlypositiveinthe

mediumterm,and61%beingsignificantlypositiveinthelongerterm.

Column2showsthedistributionofsignandsignificanceforthesubsetofstudies

thatusepost-programemploymentratestoevaluatetheALMPprogram.These111

studiesaccountfor490programestimates(57%ofourtotalsample).Theshortterm

programestimatesfromthissubsetofstudiesaresomewhatlesspositivethaninthe

overallsample.Inthemediumandlongerterms,however,thediscrepancydisappears.

Asdiscussedbelow,thesepatternsarenotexplainedbydifferencesinthetypesof

ALMPprogramsanalyzedindifferentstudies,orbydifferencesinparticipant

11

characteristics.Instead,theyreflectatendencyforstudiesbasedonmodelsofthetime

tounemploymentexit(whichareincludedincolumn1butexcludedincolumn2)to

exhibitmorepositiveshorttermimpactsthanstudiesbasedonemployment.

Column3ofTable2showsthedistributionsofsignandsignificanceassociated

withtheestimatedemploymenteffectswherewecanextractbothanactualprogram

effect(typicallythecoefficientfromalinearprobabilitymodel)andtheemployment

rateofthecomparisongroup.14Thedistributionsareverysimilartothoseincolumn2,

suggestingthatthereisnosystematicbiasassociatedwiththeavailabilityofanimpact

effectandthecomparisongroupemploymentrate.

Finally,columns4and5reportthemeanandmedianofthedistributionsof

estimatedprogrameffectsforthesubsampleincolumn3.Theshortrunprogram

effectsarecenteredjustabovezero,withameanandmedianof1.6ppt.and1.0ppt.,

respectively.Inthemediumtermthedistributionshiftsrightbutalsobecomesslightly

moreasymmetric,withameanandmedianof5.4and3.0ppt.,respectively.Inthelong

termthereisafurthershiftright,particularlyintheupperhalfofthedistribution,witha

meanandmedianof8.7ppt.and4.9ppt.,respectively.

Positiveskewinthedistributionofestimatedeffectsisofteninterpretedinthe

metaanalysisliteratureasevidenceof"publicationbias",particularlyifthepositive

effectsareimpreciselyestimated(seee.g.,StanleyandDoucouliagos,2012).Some

insightintothisissueisofferedbythe"forestplots"inFigures2a,2b,and2c,which

showthecumulativedistributionsofprogramestimatesateachtimehorizon,along

withbandsrepresentingthestandarderrorsoftheestimates.15

Inspectionofthesegraphsconfirmsthattheoveralldistributionofprogram

effectsshiftstotherightasthetimehorizonisextended.Atallthreehorizonsthereis14Inmanycasesastudyreportstheimpactontheemploymentrateoftheprogramgroupbutdoesnotreporttheemploymentrateofthecomparisongroup.Asdiscussedbelow,weneedthelatternumbertoconstructeffectsizesorproportionalimpactsontheemploymentrate.15Thedistributionsarelimitedtoestimatesforwhichwealsohaveanestimateoftheassociatedstandarderror.InformationonthestandarderrorsofprogramestimateswasnotextractedinCKW.Thus,theestimatesarefromthelateststudiescollectedinoursecondround.

12

alsosomepositiveskewinthedistributionofestimatedeffects.Interestingly,however,

theconfidenceintervalsdonotappeartobesystematicallywiderforestimatesinthe

upperorlowertailsofthedistribution.Instead,thereareahandfulofpositiveoutliers

intheshortandmediumtermdistributionsthatpushtheunweightedmeanabovethe

medianandtheprecision-weightedmean.

ReturningtoTable2,columns4and5alsoshowthemeanandmedianprogram

effectsforestimatesthatareclassifiedassignificantlypositive,insignificant,or

significantlynegative.Aswouldbeexpectedifdifferencesinsignandsignificanceare

mainlydrivenbydifferencesinthemagnitudeoftheprogramestimates–ratherthanby

differencesinthestandarderrorsoftheestimates–themeanandmedianarelargeand

positiveforsignificantpositiveestimates,largeandnegativeforsignificantnegative

estimates,andclosetozeroforinsignificantestimates.Thispatternisillustratedin

AppendixFigures2a,2b,and2c,whereweplotthehistogramsofestimatedeffectsat

eachtimehorizon,separatingtheestimatesbycategoryofsignandsignificance.Atall

threetimehorizons,thesubgroupsofestimatesappeartobedrawnfromdistributions

thatarecenteredondifferentmidpoints.Thisseparationsuggeststhatthesignand

significanceofanestimatecanserveasnoisyindicatoroftheunderlyingeffect.

c.VariationinAverageProgramImpacts

Tables3aand3bprovideafirstlookatthequestionofhowaverageALMP

impactsvaryacrossdifferenttypesofprogramsanddifferentparticipantgroups.For

eachsubsetofestimatesweshowthemeanprogrameffectsateachtimehorizonand

thecorrespondingfractionofprogramestimatesthatissignificantlypositive.

Focusingfirstoncomparisonsacrossprogramtypes(Table3a),noticethat

trainingandprivatesectoremploymentprogramstendtohavesmallaverageeffectsin

theshortrun,coupledwithmorepositiveaverageimpactsinthemediumandlonger

runs.Incontrast,JSAprogramsandALMPsinthe"other"categoryhavemorestable

impacts.Theseprofilesareconsistentwiththenatureofthetwobroadgroupsof

13

programs.Participantsintrainingandprivatesubsidyprogramsoftensuspendtheir

normaljobsearcheffortsanddevotetheirtimetoprogramactivities--aso-called"lock-

in"effectthattypicallyleadstoworseoutcomesintheimmediatepost-programperiod

(seee.g.,HamandLalonde,1996).16Assumingthatinvestmentsmadeduringthe

programperiodarevaluable,however,theoutcomesofparticipantswillgraduallycatch

upwiththoseofthecomparisongroup.17Bycomparison,JSAprogramsandother

programsthatincludemonitoringofsearcharedesignedtopushparticipantsintothe

labormarketquickly,withlittleornoinvestmentcomponent.Intheabsenceoflarge

returnstorecentjobexperience,itisunlikelythattheseprogramscanhavelargelong

runeffects.18

AnotherclearfindinginTable3aistherelativelypoorperformanceofpublic

sectoremploymentprograms–aresultthathasbeenfoundinotherpreviousanalyses

(e.g.,Heckmanetal.,1999,andCKW).

AppendixFigure3ashowshowtherelativeshareofdifferenttypesofALMPs

havechangedoverthe30yearperiodcoveredbyoursample.Thesharesoftrainingand

JSAprogramsisrelativelystable,whiletheshareofpublicsectoremploymentprograms

hasfallensharply,perhapsreflectingthemorenegativeevaluationresultsthatthese

programshaveoftenreceived.

AppendixFigure3bshowsthevariationovertimeinourtwomeasuresof

programimpact.Overall,thesignandsignificanceclassificationsofshortterm,medium

termandlongtermestimatesarequitestableovertime,withlittleindicationthatmore

recentprogramsaremoreorlesslikelytoshowsignificantpositiveresults.Thereis

16IncaseswheretheprogramgroupisdrawnfromtheregularUIsystem,participantsintrainingandsubsidizedemploymentopportunitiesareoftenexemptfromsearchrequirementsthatareimposedonnon-participants--seee.g.Biewenetal.(2014)foradiscussionintheGermancontext.17AsnotedbyMincer(1974)asimilarcross-overpatternisobservedinthecomparisonofearningsprofilesofhighschoolgraduatesandcollegegraduates.18Evidenceonthevalueoflabormarketexperienceforlowerskilledworkers(GladdenandTaber,2000;CardandHyslop,2005)suggeststhatthereturnsaremodestandunlikelytoexceed2or3percentperyearofwork.

14

morevariabilityinthemeanimpactsontheprobabilityofemployment,withsome

evidenceofanupwardtrend,particularlyfortheshortandmediumtermimpacts.

ThemiddlerowsofTable3bcomparethedistributionsofprogrameffectsby

participantagegroupandgender.TheresultsforPPSswhichincludeallagegroupsare

quitesimilartotheresultsfortheoverallsample,whiletheresultsforyouth

participantsshowamixedpattern,withrelativelysmallaverageprogrameffectson

employmentatalltimehorizons(columns4-6),butmoreevidenceofpositivelong-run

impactsbasedonsignandsignificance(columns7-9).Thedifferencesacrossgender

groupsaremoresystematicandindicatethataverageestimatedprogrameffectsare

slightlylargeratalltimehorizonsforfemales(columns4-6)andhaveahigher

probabilityofbeingsignificantlypositive(columns7-9).

Finally,thebottomrowsofTable3bcontractsresultsfromevaluationsbasedon

randomizeddesignsandnon-experimentaldesigns.Thecomparisonsofmeaneffects

suggestthatexperimentallybasedestimatestendtobelargerintheshortrunand

declineovertime,whereasnon-experimentallybasedestimatestendtobecomelarger

(morepositive)overtime.Wecautionthatthesesimple"oneway"contrastsmustbe

interpretedcarefully,however,becausetherearemultiplesourcesofpotential

heterogeneityintheprogramimpacts.Forexample,manyoftheexperimental

evaluationsfocusonJSAprograms,whereasmanyofthenon-experimentalevaluations

focusontrainingprograms.ThemetaanalysismodelsinSectionIVdirectlyaddressthis

issueusingamultivariateregressionapproach.

d.ProfileofPost-ProgramImpacts

Simplecomparisonsacrosstheimpactestimatesinoursamplesuggestthat

ALMPshavemorepositiveaverageeffectsinthemediumandlongerterms.Toverify

thatthisisactuallytrueforagivenprogramandparticipantsubgroup–andisnot

simplyanartifactofheterogeneityacrossstudies–weexaminethewithin-PPS

evolutionofimpactestimatesinTable4.

15

Columns1-3showthechangesinestimatedprogrameffectsontheprobability

ofemploymentforthesubsetofstudiesforwhichweobservebothshortandmedium

termestimates,mediumandlongtermestimates,andshortandlongtermestimates,

respectively.Consistentwiththesimplecross-sectionalcomparisons,thewithin-PPS

effectstendtoincreaseasthetimehorizonisextendedfromtheshortruntothe

mediumrun,orfromtheshortruntothelongrun.Theaveragechangebetweenthe

mediumandlongerrunsisessentiallyzero.

Comparingacrossprogramtypesitisclearthatthepatternofrisingimpactsis

drivenbytrainingprograms,whichshowarelativelylargegaininestimatedprogram

effectsfromtheshorttermtothemediumterm.Thepatternsfortheothertypesof

programssuggestrelativelyconstantordecliningaverageprogrameffectsoverthe

post-programtimehorizon.Inparticular,incontrasttothepatternsinTable3a,thereis

noindicationofariseinimpactsforprivateemploymentsubsidyprogramsovertime,

suggestingthatthegainsinTable3amaybedrivenbyheterogeneitybetweenstudies.

Wereturntothispointbelow.

Incolumns4-6weexaminethewithin-studychangesinsignandsignificancefor

abroadersetofstudies.Here,weassignavalueof+1toPPSestimatesthatchange

frominsignificanttosignificantlypositiveorfromsignificantlynegativetoinsignificant;

-1toestimatesthatchangefromsignificantlypositivetoinsignificantorfrom

insignificanttosignificantlynegative;and0toestimatesthathavethesame

classificationovertime.Thissimplesummarypointstosimilarconclusionsasthe

changesinestimatedprogrameffects,thoughJSAprogramsshowmoreevidenceofa

riseinimpactsfromtheshort-runtothemediumrunincolumn4thanthecomparison

ofestimatedeffectsontheprobabilityofemploymentincolumn1.

AppendixTables1aand1bpresentfullcross-tabulationsofsign/significanceat

thevariouspost-programtimehorizons.Assuggestedbythesimpleclassification

systemusedinTable4,mostprogramestimateseitherremaininthesamecategory,or

becomemorepositiveovertime.

16

IV.MetaAnalyticModelsofProgramImpacts

a.ConceptualFramework

ConsideranALMPevaluationthatmodelsanoutcomeyobservedformembers

ofbothaparticipantgroupandacomparisongroup.Letbrepresenttheestimated

impactoftheprogramontheoutcomesoftheparticipantsfromagivenevaluation

design,andletβrepresenttheprobabilitylimitofb(i.e.,theestimatethatwouldbe

obtainedifthesamplesizefortheevaluationwereinfinite).Understandardconditions

theestimatebwillbeapproximatelynormallydistributedwithmeanβandsomelevel

ofprecisionPthatdependsonboththesamplesizefortheevaluationandthedesign

featuresofthestudy.19Thereforewecanwrite:

b=β+P─1/2z, (1)

wherezisarealizationfromadistributionthatwillbeclosetoN(0,1)ifthesamplesize

islargeenough.ThetermP─1/2zhastheinterpretationoftherealizedsamplingerror

thatisincorporatedinb.

Assumethatthelimitingprogrameffectassociatedwithagivenstudy(β)canbe

decomposedas:

β=Xα+ε. (2)

whereαisavectorofcoefficientsandXcapturestheobservedsourcesof

heterogeneityinβ,arisingforexamplefromdifferencesinthetypeofprogramorthe

genderorageoftheprogramparticipants.Thetermεrepresentsfundamental

heterogeneityinthelimitingprogrameffectarisingfromtheparticularwayaprogram

wasimplemented,orspecificfeaturesoftheprogramoritsparticipants,orthenature

ofthelabormarketenvironment.

19Forexample,inanexperimentwith50%ofthesampleinthetreatmentgroupandnoaddedcovariates,P=N/[2σ2(1+δ2)],whereNisthesamplesize,σisthestandarddeviationoftheoutcomeyinthecontrolgroup,andδσisthestandarddeviationoftheoutcomefortheprogramgroup.Inmorecomplexdesignssuchasdifferenceindifferencesorinstrumentalvariablestheprecisionwillbesmaller.

17

Equations(1)and(2)leadtoamodelfortheobservedprogramestimatesofthe

form:

b=Xα+u, (3)

wheretheerroru=ε+P─1/2zincludesboththesamplingerrorintheestimateband

theunobserveddeterminantsofthelimitingprogrameffectforagivenstudy.Weuse

simpleregressionmodelsbasedonequation(3)toanalyzetheprogrameffectsonthe

probabilityofemploymentthatareavailableinoursample.Weinterpretthesemodels

asprovidingdescriptivesummariesofthevariationinaverageprogrameffectswith

differencesintheobservedcharacteristicsofagivenprogramandparticipantgroupin

oursample.Recognizingthestructureoftheerrorcomponentin(3)wepreferOLS

estimation,whichweightseachestimatedprogrameffectequally,ratherthanprecision-

weighedestimation,whichwouldbeefficientundertheassumptionthatε=0.20Aswe

showbelow,incontrastto“classical”metaanalysissettingswhereeachestimateis

basedonaclinicaltrialofthesamedrug,thevariationinεappearstobeparticularly

largeforALMP’s,reflectingthewiderangeoffactorsthatcanpotentiallycausea

programtobemoreorlesssuccessful.

Forourfullsampleofprogramestimatesweuse(unweighted)orderedprobit

(OP)modelsforthe3-wayclassificationofsignandsignificanceofeachestimate.Note

thatthet-statisticassociatedwiththeestimatedimpactbisjusttheratioofthe

estimatetothesquarerootofitsestimatedsamplingvariance(whichistheinverseof

itsestimatedprecision).Usingequation(3),wecanthereforewrite:

t=P1/2b

=P1/2Xα+z+P1/2ε.

IftheprecisionPoftheestimatedprogrameffectsisconstantacrossstudiesandthere

arenounobserveddeterminantsofthelimitingprogrameffect(i.e.,ε=0)thet-statistic

willbenormallydistributedwithmeanXα'whereα'=P1/2α.Inthiscasethe

20SeeSolon,HaiderandWooldridge(2015)foradiscussionofweighting.

18

coefficientsfromanOPmodelforwhetherthetstatisticislessthan-2,between-2and

2,orgreaterthan2(i.e.,thesignandsignificanceoftheestimatedprogrameffects)will

bestrictlyproportionaltothecoefficientsobtainedfromaregressionmodelofthe

correspondingestimatedprogrameffects.

Inoursampletheestimatedprecisionoftheprogramestimatesvarieswidely

acrossstudies,andthereisclearlyunobservedheterogeneityintheimpacts.21

Surprisingly,however,forstudiesthatexaminetheprobabilityofemploymentasan

outcometheestimatedcoefficientsfromOLSmodelsbasedonequation(3)andOP

modelsforsign/significanceareverynearlyproportional,suggestingthatthesame

observablefactorsthattendtoraisetheestimatedprogrameffectsalsotendtoleadto

morepositivetstatistics.Ourinterpretationofthispatternisthatthesamplingerror

componentoftheprogramestimatesissmallrelativetothevariationduetoobserved

andunobservedheterogeneity,sothet-statisticvariesacrossstudiesinproportionto

therelativemagnitudeoftheestimatedprogrameffect.WethereforeusetheOP

modelstosummarizethebroadersetofprogramestimates.

b.BasicModelsforProgramEffectandforSignandSignificance

Table5presentstheestimatesfromaseriesofregressionmodelsforoursample

ofestimatedprogrameffectsontheprobabilityofemployment.Wepooltheeffectsfor

differentpost-programhorizonsandincludedummiesindicatingwhethertheestimate

isforthemediumorlongterm(withshorttermestimatesintheomittedgroup).The

basicmodelincolumn1includesonlythesecontrolsandasetofdummiesforthetype

ofprogram(withtrainingprogramsintheomittedcategory).Consistentwiththesimple

comparisonsinTable3a,wefindthattheprogramestimatesarelargerinthemedium

21AscanbeseenfromthevaryingwidthsoftheconfidenceintervalsinFigures2a,2band2c,theprecisionoftheestimatedprogrameffectsvarieswidelyacrossstudiesinoursample.Theprecisionisessentiallyuncorrelatedwiththesamplesizeoftheevaluation(correlation=─0.02),suggestingthatstudieswithlargersamplesizeshavemorecomplexeconometricdesignsthatoffsetanypotentialgainsinprecision.

19

andlongrun,andthatpublicsectoremploymentprogramsareassociatedwithsmaller

programeffects.

Themodelincolumn2introducesadditionalcontrolsforthetypeofparticipant

andstudycharacteristics,whicharereportedinTable7anddiscussedbelow.These

controlsslightlyattenuatethegrowthinprogrameffectsoverlongerpost-program

horizonsandalsoreducethemagnitudeoftheJSAprogrameffectfrom-3.2ppts.(and

significant)to-0.1ppts(andinsignificant).

Columns3-5introduceaparallelsetofmodelsthatallowthetimeprofilesof

post-programimpactstovarywiththetypeofprogram.Inthesespecificationsthe

"maineffects"foreachprogramtypeshowtheshorttermimpactsrelativetotraining

programs(theomittedtype),whiletheinteractionsofprogramtypewithmediumterm

andlongtermdummiesshowhowtheimpactsevolverelativetotheprofilefortraining

programs(whicharesummarizedbythemaineffectsinthefirsttworows).Wepresent

modelswithandwithoutadditionalcontrolsincolumns3and4,andamodelwith

dummyvariablesforeachparticipant/programsubgroupincolumn5.Inthelatter

specificationthe"maineffects"forthetypeofprogramareabsorbedbythePPSfixed

effects,butwecanstillestimatethecoefficientsformediumandlongtermeffects--

whichnowmeasuretheevolutionoftheprogrameffectsforthesamePPSover

differenttimehorizons--aswellasinteractionsofthetimehorizondummieswiththe

typeofprogram.

Threekeyconclusionsemergefromthesemoreflexiblespecifications.First,as

suggestedbythepatternsinTable4,theprogramimpactsfortrainingprogramstendto

riseovertime,whiletheeffectsforjobsearchassistanceprogramsandotherprograms

(whichareobtainedbyaddingtheprogram-type/timehorizoninteractionstothetime

horizoneffectsinrows1and2)areroughlyconstant.22Second,inthemodelswithout

22Toaidintheinterpretationoftheinteractedestimates,AppendixTable4presentstheimpliedmeanprogrameffectsbyprogramtypeandtimehorizonforthemodelsincolumns3and4,andtheassociatedstandarderrors.Notethatbyconstructionthemodelincolumn3reproducesthemeansreportedin

20

PPSfixedeffects(columns3and4)theimpliedprofileofimpactsforprivatesector

employmentprogramsisrelativelysimilartotheprofilefortrainingprograms.When

thePPSeffectsareadded,however,theinteractionsbetweenprivatesectorprograms

andbothmediumtermandlongtermhorizonbecomerelativelylargeandnegative--

similartotheinteractioneffectsforJSAandotherprograms.Athirdconclusionisthat

publicsectoremploymentprogramsappeartoberelativelyineffectiveatalltime

horizons.

WehavealsoestimatedmodelssimilartothespecificationsinTable5,butusing

twoalternativemeasuresofprogramimpacts:theestimated"effectsize"(the

estimatedeffectontheemploymentrateofparticipantsdividedbythestandard

deviationofemploymentratesinthecomparisongroup),andtheproportionalprogram

effect(theestimatedeffectforparticipantsdividedbythemeanemploymentrateof

thecomparisongroup).ThesespecificationsarereportedinAppendixTables2aand2b,

respectively,andyieldverysimilarconclusionstothemodelsinTable5.Essentially,

thesealternativechoicesleadtorescalingofthecoefficientsofthemetaanalysis

modelswithverysmallchangesintherelativemagnitudesofdifferentcoefficients.

AlimitationoftheanalysisinTable5isthatestimatedprogrameffectsareonly

availablefor40%ofouroverallsample.Tosupplementthesemodelsweturnto

orderedprobitmodelsforsignandsignificance.Thefirst4columnsofTable6presenta

seriesofOPmodelsthatareparalleltothoseinTable5,butfittoouroverallsampleof

programestimates.Thespecificationsincolumns1and3havenocontrolsotherthan

dummiesformediumandlongtermhorizonsandthetypeofALMP--inthelattercase

interactingthetypeofprogramwiththetimehorizondummies.Columns2and4

reportexpandedspecificationsthataddthecontrolvariablesreportedinTable7.

Column5ofTable6repeatsthespecificationfromcolumn4,butfittothesubsampleof

columns4-6ofTable3a.Forthespecificationincolumn4ofTable5wenormalizethecovariatestohavemean0andfitthemodelwithoutanintercept:thusthemeanprogrameffectsareinterpretedasmeansforaprogramandparticipantgroupwiththemeancharacteristicsofoursample.

21

352programestimatesforwhichwehaveanestimateoftheprogrameffectonthe

probabilityofemployment.Themodelincolumn6reproducesthespecificationin

column3,butaddingPPSfixedeffects.AsinthemodelinthelastcolumnofTable5,

thecoefficientsinthisspecificationmeasuretheevolutionofthefactorsdetermining

signandsignificancewithinagivenPPS.Finally,column7presentsestimatesfroma

linearregressionreplacingthecategoricaloutcomevariablewithvaluesof-1,0,and+1,

alsoincludingPPSfixedeffects.

TheOPmodelsinTable6yieldcoefficientsthatareveryhighlycorrelatedwith

thecorrespondingcoefficientsfromtheprogrameffectmodelsinTable5,butroughly

10timesbiggerinmagnitude.Forexample,thecorrelationofthe14coefficientsfrom

thespecificationincolumn4ofTable6withcorrespondingcoefficientsfromthe

specificationincolumn4ofTable5is0.84.23InparticulartheOPmodelsconfirmthat

theimpactsofjobsearchassistanceandotherprogramstendtofaderelativetothe

impactsoftrainingprograms,andthatpublicsectoremploymentprogramsare

relativelyineffectiveatalltimehorizons,regardlessofhowtheoutcomesaremeasured

intheevaluation.24

ThemodelsincludingPPSfixedeffectsincolumns6(orderedprobit)and7

(linearregression)ofTable6alsoimplythesamequalitativefindings,indicatingthat

withinagivenPPStheestimatedprogrameffectbecomesmorepositiveinthelonger

run.Thecoefficientsofthelinearregressionmodelarescaledbyafactorof

approximatelyfiverelativetotheorderedprobitspecification.

c.ParticipantandStudyCharacteristicsintheBasicModels

23Theregressionmodelis:OP-coefficient=-0.02+10.57×Effect-coefficient,R-squared=0.70.24Wealsofittwosimplerprobitmodelsfortheeventsofreportingapositiveandsignificantornegativeandsignificantestimate,reportedinAppendixTable3.Aswouldbeexpectediftheorderedprobitspecificationiscorrect,thecoefficientsfromthemodelforasignificantlypositiveeffectarequiteclosetotheOPcoefficients,whilethecoefficientsfromthemodelforasignificantlynegativeeffectarecloseinmagnitudebutoppositeinsign.

22

Theestimatedcoefficientsfortheextracontrolvariablesincludedinthemodels

incolumns2,4ofTable5andcolumns2,4,and5ofTable6arereportedinTable7.The

coefficientestimatesfromthetwomodelsfortheeffectsontheprobabilityof

employment(columns1,2)arequitesimilarandsuggestthattheimpactofALMPsvaries

systematicallywiththetypeofparticipant(withlargereffectsforthelongterm

unemployed),theiragegroup(morenegativeimpactsforolderandyounger

participants),andtheirgender(largereffectsforfemales).Theestimatedprogram

effectsarealsosomewhatlargerforstudiesestimatedonGerman,AustrianorSwiss

data,buttherearenolargeorsignificantdifferencesacrosstheothercountrygroups.

ThecoefficientsfromtheOPmodels(columns3-4)confirmmostofthese

conclusionsaboutthedifferentialimpactsofALMPsacrossdifferentparticipantgroups

anddifferentcountries.25Inparticular,theOLSmodelsfortheprogrameffectsonthe

probabilityofemploymentandtheOPmodelsforsignandsignificanceshowsmaller

impactsforyoungparticipantsandolderparticipants,relativetotheimpactsonmixed

agegroups,andlargerimpactsforlong-termunemployedparticipants.TheOPmodels

fittotheoverallsample(columns3,4)alsopointtoalargerpositiveimpactfor

disadvantagedparticipantsrelativetoUIrecipients,whereastheprogrameffectmodels

andtheOPmodelsfittotheprogrameffectsubsample(column5)yieldaninsignificant

coefficient,arguablyduetothesmallnumberofstudiesthatfocusonthisgroup.26

25Thecorrelationbetweenthecoefficientsincolumns2and4ofTable7is0.69.26ApotentiallyrelevantdimensionofprogrameffectivenessconcernsthetimeALMPparticipantshavespentinunemploymentbeforeenteringtheprogram.Biewenetal.(2014)investigatethisissueusingthreestratatoestimatetreatmenteffects:1-3months,4-6months,and7-12monthsofunemployment,respectively.Forlongertrainingprograms(meandurationof226days)theestimatedshort-termimpactsandstandarderrorsforthethreestrataare0.00(0.04),0.00(0.04),0.08(0.40)formales,and0.06(0.03),-0.01(0.045),-0.04(0.02)forfemales;medium-termimpactsare0.05(0.03),0.07(0.04),0.09(0.045)formales,and0.06(0.03),0.11(0.05),0.09(045)forfemales.Theseresultsshowsomeindicationthatprogramparticipantsfromstratawithlongerelapsedunemploymentdurationsbenefitmorethanothergroups.Thisresultisinlinewiththefindingsfromourstratificationofprogramintakegroupintoshort-termunemployed(“UIrecipients”),long-termunemployed,andparticipantswithoutbenefitentitlement(“disadvantaged”).Amoredetailedinvestigationofthisissuewithinthemeta-analysisframeworkisnotpossible,sincetheprimarystudiesrarelyreportinformationonelapsedtimeinunemploymentbeforeprogramstart.

23

OnenotabledifferencebetweentheprogrameffectmodelsandtheOPmodels

concernstherelativeimpactofALMPsonfemaleparticipants.IntheOLSprogrameffect

modelstheestimatedcoefficientsforfemaleparticipantsarearound0.04to0.05in

magnitude,andstatisticallysignificantatconventionallevels(withtstatisticsaround2).

IntheOPmodels,bycomparison,thecorrespondingcoefficientsarerelativelysmallin

magnitudeandfarfromsignificant.Furtherinvestigationrevealsthatthisdivergenceis

drivenbytheuppertailofprogrameffectestimatesforfemaleparticipants(see

AppendixFigure4),andinparticularbytherelativelylargeestimatedeffectsforfemale

PPSsthatshowasignificantpositiveeffect.27Thisuppertaildoesnotappeartobe

drivenbyafewoutliers,butinsteadreflectsasystematicallyhigherprobabilityof

estimatingalargepositiveeffectwhentheparticipantgroupislimitedtofemales.28

AninterestingaspectoftheOPmodelsisthepatternofcoefficientsassociated

withthechoiceofdependentvariable,reportedinthetoprowsofTable7.These

coefficientsshowthatstudiesmodelingthehazardrateofexitingthebenefitsystemor

theprobabilityofunemploymentaresignificantlymorelikelytoreportpositivefindings

thanstudiesmodelingemployment(theomittedcategory)orearnings.29

ThemodelssummarizedinTable7alsocontrolforthedurationoftheprogram,

usingasimpledummyvariableforwhethertheprogramlastedlongerthan9months.

Programdurationcanbeseenasaroughproxyforthecostoftheprogram,as

informationoncosteffectivenessisverysparseinmostofthesurveyedstudies.Our

estimatesdonotindicateanysystematicadvantageforlongerprograms--indeed

acrossallthespecificationsthecoefficientisnegative,thoughstatisticallyinsignificant.

27Themedianand75thpercentilesoftheeffectsizedistributionforfemaleparticipantgroups,conditionalonapositiveimpact,are0.25,and0.46respectively.Bycomparison,thecorrespondingstatisticsforthecombinedmaleandmixedgenderparticipantgroupsare0.15,and0.27.28Wealsoestimatedseparateprogrameffectmodelsfordifferenttypesofparticipants--thosefromtheregularUIsystemversuslongtermunemployedordisadvantagedgroups.WefoundasignificantpositivecoefficientforfemaleparticipantsinthemodelsforbothUIrecipientsandthelongtermunemployed.29Estimatesfrominteractedmodelsthatallowdifferenteffectsofthedependentvariableatdifferenttimehorizons(notreportedinthetable)showthatthepositivecoefficientassociatedwiththeuseofexithazardsislargelyconfinedtoshorttermimpacts.

24

d.RandomizedversusNon-ExperimentalDesigns

AlongstandingconcernintheALMPliterature(e.g.,Ashenfelter,1987)isthe

unreliabilityofnon-experimentalestimators.Thisconcernledtoaseriesoflargescale

experimentsintheU.S.designedtotestjobtrainingprograms(Bloometal.,1997;

Schochet,Burghardt,andGlazerman,2001)andalargeliteraturecomparing

experimentalandnon-experimentalestimatesforthesameprogram(e.g.,Lalonde,

1986;SmithandTodd,2005).

Onesimplewaytoassesstheseconcernsistocomparethemagnitudesofthe

estimatedeffectsfrompapersbasedonexperimentaldesignstothosefromnon-

experimentaldesigns.Todothis,weincludeasimpleindicatorforanexperimental

designinthemodelsinTable7.Intheprogrameffectmodels(columns1and2)the

coefficientofthedummyisverysmallinmagnitude(lessthan1.0ppt)and

insignificantlydifferentfromzero.Likewise,inthemainOPmodels(columns3-4)the

coefficientissmallinmagnitude.ItisalittlelargerinmagnitudewhentheOPmodels

arerestrictedtothesetofstudieswithestimatedprogrameffects(column5)but

relativelyimprecise,perhapsreflectingtherelativelylargenumberofparameters

includedinthemodelrelativetothesamplesizeortheunevendistributionof

experimentalevaluationsacrossprogramtypes.Weconcludefromthisanalysisthat

thereislittleornoevidencethatresultsfromexperimentally-baseddesignsinthe

recentALMPliteratureare"lesspositive"or"lesssignificant"thanresultsfromnon-

experimentaldesigns.

e.PublicationBiasandp-hacking

Arelatedconcern,widelydiscussedinthemetaanalysisliterature(e.g.,

Rothstein,Sutton,andBorenstein,2005)isthatthesetofestimatedprogramimpactsin

theavailableliteraturecontainasystematicpositivebias,eitherbecauseanalystsonly

writeupandcirculatestudiesthatshowapositiveeffect(so-calledfiledrawerbias)or

25

becausetheychoosespecificationsthattendtoyieldpositiveandsignificanteffects(so-

calledp-hacking).

Astandardwaytolookforevidenceofpublicationbiasistoexaminefunnel

plotsoftherelationshipbetweentheestimatedprogrameffectsandtheirprecision

(Suttonetal.,2000).Figures3a,3b,and3cpresenttheseplotsfortheprogrameffects

foremploymentinoursample,restrictingattention(asinFigures2a-2c)toestimates

thathaveacorrespondingsamplingerroravailable.Forreference,wealsoshowthe

boundariesofthe“t=2”relationshipineachgraph.30Contrarytotheinvertedfunnel

patterntypicallyuncoveredinthemetaanalysisliterature(e.g.,Doucouliagosand

Stanley,2009;Havranek,2015;WolfsonandBelman,2015),thereisalotofdispersion

intheestimatedprogrameffectsatallthreetimehorizons,evenamongstudieswith

highlevelsofprecision,suggestingthatthevariationintheestimatesisnotjustaresult

ofsamplingerror.

Amoreformaltestforpublicationbiasistoregresstheestimatedprogram

effectfromagivenstudyandspecificationontheassociatedsamplingerrorofthe

estimateandotherpotentialcontrolvariables.UsingthenotationofSectionIVa,the

regressionmodelis:

b=Xα+θP─1/2+ν (4)

whereνrepresentsaresidual.Theestimateofθisinterpretedasatestforasymmetry

inthefunnelplotrelationshipbetweentheestimatedprogrameffectsandtheir

precision:ifthesamplecontainsmoreimpreciselyestimatedlargepositiveeffectsthan

largenegativeeffects,θwillbepositive.Stanley(2008)suggeststhatthemodelbe

estimatedbyweightedleastsquares,usingtheprecisionofeachestimate(i.e.,its

inversesamplingvariance)asaweight.Ifsamplingerroristheonlysourceofresidual

variationin(4)thiswillleadtoefficientestimates.

30Sincet=P1/2b,the“t=2”relationshipisP=4/b2whichisapairofhyperbolascenteredaroundtheyaxisinafunnelplot.

26

EstimationresultsforthismodelarepresentedinTable8.Wepresent

estimatesfromfourspecificationsestimatedbyunweightedOLSandprecision-weighted

leastsquares.31Theresultsgivenoindicationofpublicationbias.Theunweighted

estimateincolumn3,forexample,whichisbasedonamodelthatincludesourrichest

setofcontrols,impliesthatastudywitha1percentagepointlargerstandarderrorfor

theestimatedprogrameffectontheprobabilityofemploymentwillhaveabouta0.01

percentpointlargerestimatedprogrameffect.Thecorrespondingweightedestimate

suggestsaslightlylarger0.03percentpointlargerestimate,butislessprecise.

Wesuspectthatthereareatleasttwoexplanationsfortheapparentlackof

publicationbiasintheALMPliterature.ThefirstisthatALMPevaluationsareoften

conductedincloseco-operationwiththeagenciesthatoperatetheprograms.Insuch

settingsresearcherscannotsimplyshelvepapersthatshowsmallor"wrongsigned"

impacts.Theymayalsofindithardtochooseamongspecificationstoobtainamore

positiveprogrameffect.Asecondfactoristhatrefereesandotherresearchershaveno

strongpresumptionthatALMP'snecessarily"work",orthatafindingofanegativeor

insignificanteffectisuninteresting,sincemanyimportantpapersinthefield(e.g.,

Lalonde,1986;HeckmanandHotz,1989)reportsmallornegativeimpactsofALMPs.

Ourmodelsalsocontrolfortwootherfeaturesofastudythatmaybe

informativeaboutthepresenceofpublicationbias:whetheritwaspublished,andthe

numberofcitationsitreceived(measuredfromaGoogleScholarsearchinSpring

2015).32Thecoefficientsassociatedwithbothvariables(showninTable7)aresmall

andinsignificantacrossallspecifications,confirmingthatthereisnotendencyformore

positivestudiestobepublishedortobemorehighlycited.

f.AreSomeProgramsBetter(orWorse)forDifferentParticipantGroups?

31Theestimatedprecisionoftheestimatesinoursamplerangesfrom0.026toover50,000.TostabilizetheestimatesweWinsorizetheprecisionweightsatthe10thand90thpercentiles.32Toaccountforlagsinthecitationprocesswemodelcitationsastherankwithinthedistributionofcitationsforpaperswritteninthesameyear.

27

AlongstandingquestionintheALMPliteratureiswhethercertainparticipant

groupsare"bettermatched"tospecifictypesofprograms(forananalysisinthe

GermancontextseeBiewenetal.2007).WeaddressthisinTable9,whichpresents

separatemodelsfortheprogrameffectsfromdifferenttypesofALMPs.

Asabenchmarkcolumn1presentsabaselinespecificationfittoall5program

types,withdummiesfortheprogramtypes(notreported)andcontrolsfortheintake

group,thegendergroup,andtheagegroup.33The(omitted)basegroupiscomprisedof

mixedgenderandagegroupsfromtheregularUIrolls.Inthispooledspecificationthe

estimatedeffectsforfemalesandlongtermunemployedparticipantsaresignificantly

positive,whilethecoefficientforolderparticipantsissignificantlynegative,andthe

coefficientforyoungparticipantsisnegativeandmarginallysignificant.

Columns2-6reportestimatesforthesamespecification(minusthecontrolsfor

thetypeofprogram)fitseparatelytotheestimatedeffectsforeachofthe5program

types.Comparisonsacrossthesemodelssuggestthatlong-termunemployed

participantsbenefitrelativelymorefrom"humancapital"programs(i.e.,trainingand

privatesectoremployment),andrelativelylessfrom"workfirst"programs(i.e.,job

searchandotherprograms).Incontrast,disadvantagedparticipantsappeartobenefit

morefromworkfirstprogramsandlessfromhumancapitalprograms.Female

participantsalsoappeartobenefitrelativelymorefromtrainingandprivatesector

subsidyprograms,whiletherelativeeffectsforyouthsandolderparticipantsarenot

muchdifferentacrosstheprogramtypes.

Overalltheseresultssuggestthattheremaybepotentialgainstomatching

specificparticipantgroupstospecifictypesofprograms,thoughthesmallsamplesizes

formostoftheprogramtypesmustbenoted.Attemptstoexpandthepowerofthe

analysisbyusingOPmodelsforthesignandsignificanceoftheprogramestimateslead

33Thisisasimplifiedversionofthespecificationreportedincolumn2ofTable5andcolumn1ofTable7.

28

togenerallysimilarconclusionsastheprogrameffectmodelsreportedinTable9with

onlymodestgainsinprecision.

g.EffectsofCyclicalConditions

AnotherlongstandingquestionintheALMPliteratureiswhetherprogramsare

more(orless)effectiveindifferentcyclicalenvironments.34Oneviewisthatactive

programsarelesseffectiveinadepressedlabormarketbecauseparticipantshaveto

competewithother,moreadvantagedworkersforalimitedsetofjobs.Analternative

viewisthatALMPsaremoreeffectiveinweaklabormarketsbecauseemployers

becomemoreselectiveinaslackmarket,increasingthevalueofaninterventionthat

makesworkersmorejob-ready.

ThreepreviousstudieshaveinvestigatedALMPeffectivenessoverthebusiness

cycle.Kluve(2010)usesbetween-countryvariationinasmallEuropeanmetadataset,

whileLechnerandWunsch(2009)andForslundetal.(2011)analyzeprogramsin

GermanyandSweden,respectively.Allthreestudiessuggestapositivecorrelation

betweenALMPeffectivenessandtheunemploymentrate.

Toprovidesomenewevidenceweaddedtwoalternativecontextualvariablesto

ouranalysis,representingtheaveragegrowthrateofGDPandtheaverage

unemploymentrateduringtheyearsthetreatmentgroupparticipatedintheprogram.

Sincegrowthratesandunemploymentratesvarywidelyacrosscountries,wealso

introducedasetofcountrydummiesthatabsorbanypermanentdifferencesinlabor

marketconditionsacrosscountries.Theeffectofthesedummiesisinterestinginitsown

rightbecausethesharesofdifferentprogramtypesandparticipantgroupsalsovary

widelyacrosscountries,leadingtothepossibilityofbiasinthemeasuredeffectsof

programtypesandparticipantgroupsifthereareunobservedcountryspecificfactors

thataffecttheaveragesuccessofALMPsindifferentcountries.

34Arelatedquestioniswhetherprogramexternalitiesarebiggerorsmallerinweakorstronglabormarkets.ThisisaddressedintheexperimentconductedbyCreponetal.(2013).

29

TheresultsofouranalysisaresummarizedinTable10.Forreferencecolumn1

presentsabenchmarkspecificationidenticaltothesimplifiedmodelincolumn1of

Table9.Column2presentsthesamespecificationwiththeadditionof37country

dummies.Theadditionofthesedummiesleadstosomemodestbutinteresting

changesintheestimatedcoefficientsinthemetaanalysismodel.Mostnotably,the

coefficientsassociatedwithjobsearchassistance(JSA)and"other"programsboth

becomemorenegative,indicatingthattheseprogramstendtobemorewidelyusedin

countrieswhereallformsofALMPsarerelativelysuccessful.

Column3presentsamodelthatincludesthecontrolforaverageGPDgrowth

rateduringtheprogramperiod.Thecoefficientisnegativeandmarginallysignificant

(t=1.7)providingsuggestiveevidencethatALMPsworkbetterinrecessionarymarkets.A

modelthatcontrolsfortheaverageunemploymentrateshowsthesametendency

(coefficient=0.006,standarderror=0.007)thoughtheeffectislessprecise.

Aconcernwiththespecificationincolumn3isthattheaveragenumberof

programestimatespercountryissmall(manycountrieshaveonly2or3estimates)

leadingtopotentialover-fitting.Toaddressthis,weestimatedthemodelsincolumns4-

6,usingonlydatafromthefourcountriesthataccountforthelargestnumbersof

programestimates-Denmark(17estimates),France(20estimates),Germany(147

estimates)andtheU.S.(16estimates).Asshownincolumn4,ourbaselinespecification

yieldscoefficientestimatesthatarequitesimilartotheestimatesfromtheentire

sample,thoughtherelativeimpactsofJSAandotherprogramsaremorenegativein

these4countries.

Columns5and6presentmodelsthataddtheaverageGDPgrowthrateandthe

averageunemploymentrate,respectively,tothisbaselinemodel.Thesespecifications

suggestrelativelyimportantcyclicaleffectsonALMPeffectiveness.Forexample,

comparingtwosimilarprogramsoperatinginlabormarketswitha3percentagepoint

gapingrowthrates,theprogramintheslowergrowthenvironmentwouldbeexpected

tohavea0.1largerprogrameffect.

30

Toillustratethevariationdrivingtheresultsincolumns5and6,Figure4plots

theannualGDPgrowthrateinGermanyalongwiththemeanprogrameffectsin

differentyears,groupingprogramestimatesbytheyearthattheprogramwasfirst

operated.Wealsoshowthenumberofprogramestimatesforeachyear,whichvaries

substantiallyovertime.Thefigureshowsthecounter-cyclicalpatternofprogrameffects

ismainlydrivenbylargepositiveeffectsofprogramsimplementedintheearly2000s

duringaperiodofslowgrowth.

Althoughpolicymakershavetodecidewhethertoincreasespendingforactive

programsandenrollmoreparticipantsknowingonlycurrentbusinesscycleconditions,it

isalsoofinteresthowlabormarketconditionsatthetimeofprogramcompletionare

relatedtotheprogrameffects.InAppendixTable5weestimatealternative

specificationsthatcontrolfortheGDPgrowthrateandunemploymentrateatthe

beginningandaftertheendoftheprogramperiod.TheseestimatessuggestthatALMP

programstendtobeparticularlysuccessfulifparticipantsareenrolledinaprogram

duringadownturnandexittheprogramduringaperiodoffavorableeconomic

conditions.

WhiletheevidenceinTable10suggestsacountercyclicalpatternofprogram

effectiveness,itisworthemphasizingthattheexplanationforthispatternislessclear.It

ispossiblethatthevalueofagivenprogramishigherinarecessionaryenvironment.It

isalsopossible,however,thatthecharacteristicsofALMPparticipants,orofthe

programsthemselves,changeinawaythatcontributestoamorepositiveimpactina

slow-growth/high-unemploymentenvironment.

V.SummaryandConclusions

Wehaveassembledandanalyzedanewsampleofimpactestimatesfrom207

studiesofactivelabormarketpolicies.Buildingonourearlierstudy(CKW),weargue

thatitisimportanttodistinguishbetweenimpactsatvarioustimehorizonssince

completionoftheprogram,andtoconsiderhowthetimeprofileofimpactsvariesby

31

thetypeofALMP.Wealsostudytheimportanceofparticipantheterogeneity,andlook

forevidencethatspecificsubgroupsmaybenefitmoreorlessfromparticulartypesof

programs.Finally,westudyhowthestateofthelabormarketaffectsthemeasured

effectivenessofALMPs.

WithregardtotheimpactsofdifferenttypesofALMPs,wefindthatthetime

profilesof"workfirst"styleprogramsthatofferjobsearchassistanceorincentivesto

enterworkquicklydifferfromtheprofilesof"humancapital"styletrainingprograms

andpublicsectoremploymentprograms.Humancapitalprogramshavesmall(orin

somecasesevennegative)shorttermimpacts,coupledwithlargerimpactsinthe

mediumorlongerrun(2-3yearsaftercompletionoftheprogram),whereastheimpacts

fromworkfirstprogramsaremorestable.Wealsoconfirmthatpublicsector

employmentprogramshavenegligible,orevennegativeprogramimpactsatalltime

horizons.

Withregardtodifferentparticipantgroups,wefindthatfemaleparticipantsand

thosedrawnfromthepooloflongtermunemployedtendtohavelargerprogram

effectsthanothergroups.Incontrast,theprogramestimatesforyouthsandolder

workersaretypicallylesspositivethanforothergroups.Wealsofindindicationsof

potentialgainstomatchingdifferentparticipantgroupstospecificprograms,with

evidencethatworkfirstprogramsarerelativelymoresuccessfulfordisadvantaged

participants,whereashumancapitalprogramsaremoresuccessfulforthelongterm

unemployed.

Withregardtothestateofthelabormarket,wefindthatALMPstendtohave

largerimpactsinperiodsofslowgrowthandhigherunemployment.Inparticular,we

findarelativelylargecyclicalcomponentintheprogramestimatesfromfourcountries

thataccountforone-halfofoursample.Wealsofindsuggestiveevidencethathuman

capitalprogramsaremorecyclicallysensitivethanworkfirstprograms.

Ourfindingsontherelativeefficacyofhumancapitalprogramsforlongterm

unemployed,andonthelargerimpactsoftheseprogramsinrecessionary

32

environments,pointtoapotentiallyimportantpolicylesson.AsnotedbyKrueger,Judd

andCho(2014)andKroftetal.(2016),thenumberoflongtermunemployedrises

rapidlyasarecessionpersists.Thisgrouphasahighprobabilityofleavingthelabor

force,riskingpermanentlossesintheproductivecapacityoftheeconomy.Onepolicy

responseiscountercyclicaljobtrainingprogramsandprivateemploymentsubsidies,

whichareparticularlyeffectiveforthelonger-termunemployedinarecessionary

climate.

Methodologically,wefindanumberofinterestingpatternsintherecentALMP

literature.Wefindthattheestimatedimpactsderivedfromrandomizedcontrolled

trials,whichaccountforone-fifthofoursample,arenotmuchdifferentonaverage

fromthenon-experimentalestimates.Wealsofindnoevidenceof"publicationbias"in

therelationshipbetweenthemagnitudeofthepointestimatesfromdifferentstudies

andtheircorrespondingprecision.Wedofindthatthechoiceofoutcomevariableused

intheevaluationmatters,withatendencytowardmorepositiveshorttermimpact

estimatesfromstudiesthatmodelthetimetofirstjobthanfromstudiesthatmodelthe

probabilityofemploymentorthelevelofearnings.

Finally,weconcludethatmetaanalyticmodelsbasedonthesignand

significanceoftheprogramimpactsleadtoverysimilarconclusionsasmodelsbasedon

programeffects.Wearguethatthisarisesbecausemuchofthevariationinthesignand

significanceofestimatedimpactsacrossstudiesintheALMPliteratureisdrivenby

variationinestimatedprogrameffects,ratherthanbyvariationinthecorresponding

samplingerrorsoftheestimates.

33

References

Angrist,J.D.,&Pischke,J.S.(2010)."TheCredibilityRevolutioninEmpiricalEconomics:HowBetterResearchDesignisTakingtheConOutofEconometrics.JournalofEconomicPerspectives24(2):3-30.Ashenfelter,Orley(1978).“EstimatingtheEffectofTrainingProgramsonEarnings”ReviewofEconomicsandStatistics60(1):47-57.Ashenfelter,OrleyandDavidCard(1985).“UsingtheLongitudinalStructureofEarningstoEstimatetheEffectofTrainingPrograms.”ReviewofEconomicsandStatistics67(4):648-660.Ashenfelter,Orley(1987)."TheCaseforEvaluatingTrainingProgramswithRandomizedTrials."EconomicsofEducationReview6(4):333-338.Bergemann,A.H.andG.J.vandenBerg(2008)."ActiveLaborMarketPolicyEffectsforWomeninEurope-ASurvey".Annalesd'EconomieetdeStatistique91/92:385-408.Biewen,M.,B.Fitzenberger,A.Osikominu,andM.Waller(2014)" TheEffectivenessofPublicSponsoredTrainingRevisited:TheImportanceofDataandMethodologicalChoices",JournalofLaborEconomics,32:837-897.Biewen,M.,B.Fitzenberger,A.Osikominu,andM.Waller(2007)"WhichProgramforWhom?EvidenceontheComparativeEffectivenessofPublicSponsoredTrainingProgramsinGermany",IZADiscussionPaperNo.2885.Blank,CardandRobins(2000)."FinancialIncentivesforIncreasingWorkandIncomeAmongLow-IncomeFamilies."InRebeccaM.BlankandDavidCard,editors,FindingJobs:WorkandWelfareReform.NewYork:RussellSageFoundation.Bloom,HowardS.,LarryL.Orr,StephenH.Bell,GeorgeCave,FredDoolittle,WinstonLin,andJohannesM.Bos(1997).“TheBenefitsandCostsofJTPATitleII-APrograms:KeyFindingsfromtheNationalJobTrainingPartnershipActStudy.”JournalofHumanResources32(3):549–576.Card,DavidandDeanR.Hyslop(2005)."EstimatingtheEffectofaTime-LimitedEarningsSubsidyforWelfareRecipients."Econometrica73(6):1723-1770.Card,D.,J.KluveandA.Weber(2010)."ActiveLabourMarketPolicyEvaluations:AMeta-analysis."EconomicJournal120(548):F452-F477.

34

Cho,Y.andM.Honorati(2014).“EntrepreneurshipProgramsinDevelopingCountries:AMetaRegressionAnalysis.”LabourEconomics28(C):110-130.Crépon,Bruno,EstherDuflo,MarcGurgand,RolandRathelot,andPhilippeZamora(2013)."DoLaborMarketPolicieshaveDisplacementEffects?EvidencefromaClusteredRandomizedExperiment."QuarterlyJournalofEconomics128(2):531-580.Doucoulias,HristosandT.D.Stanley(2009).“PublicationSelectionBiasinMinimumWageResearch?AMeta-RegressionAnalysis.”BritishJournalofIndustrialRelations47(2):406-428.Filges,Trine,GeirSmedslund,Anne-SofieDueKnudsen,andAnne-MarieKlintJørgensen(2015)."ActiveLabourMarketProgrammeParticipationforUnemploymentRecipients:ASystematicReview."CampbellSystematicReviews2015(2).Forslund,A.,P.FredrikssonandJ.Vikström(2011).“WhatActiveLaborMarketPolicyWorksInaRecession?”.NordicEconomicPolicyReview1:171-207.Friedlander,Daniel,DavidH.GreenbergandPhilipK.Robins(1997)."EvaluatingGovernmentTrainingProgramsfortheEconomicallyDisadvantaged".JournalofEconomicLiterature,25(4):1809-1855Gladden,TriciaandChristopherTaber(2000)."WageProgressionAmongLessSkilledWorkers."InRebeccaM.BlankandDavidCard,editors,FindingWork:JobsandWelfareReform.NewYork:RussellSageFoundation.Greenberg,DavidH.,CharlesMichalopoulosandPhilipK.Robins(2003).“AMeta-AnalysisofGovernment-SponsoredTrainingPrograms.”IndustrialandLaborRelationsReview57():31-53.Grimm,M.andA.L.Paffhausen(2015).“DoInterventionsTargetedatMicro-EntrepreneursandSmallandMedium-sizedFirmsCreateJobs?ASystematicReviewoftheEvidenceforLowandMiddleIncomeCountries”.LabourEconomics32:67-85.Ham,JohnC.andRobertJ.Lalonde(1996)."TheEffectofSampleSelectionandInitialConditionsinDurationModels:EvidencefromExperimentalDataonTraining."Econometrica64(1):175-205.Havránek,Tomáš(2015)MeasuringIntertemporalSubstitution:TheImportanceofMethodChoicesandSelectiveReporting."JournaloftheEuropeanEconomicAssociation,13(6).

35

Heckman,JamesJ.,andV.JosephHotz(1989)."ChoosingAmongAlternativeNonexperimentalMethodsforEstimatingtheImpactofSocialPrograms:TheCaseofManpowerTraining."JournaloftheAmericanstatisticalAssociation84(408):862-874.Heckman,JamesJ.,HidehikoIchimura,JeffreyA.Smith,andPetraTodd(1998).“CharacterizingSelectionBiasUsingExperimentalData.”Econometrica66(5):1017-1098.Heckman,JamesJ.RobertJ.LaLondeandJeffreyA.Smith(1999)."TheEconomicsandEconometricsofActiveLaborMarketPrograms".InOrleyAshenfelterandDavidCard,editors,HandbookofLaborEconomicsVolume3a.Amsterdam:Elsevier.Hedges,LarryV.andIngramOlkin(1985).StatisticalMethodsforMeta-Analysis.Orlando:AcademicPress.Ibarrarán,P.andD.Rosas(2009),EvaluatingtheImpactofJobTrainingProgramsinLatinAmerica:EvidencefromIDBfundedoperations,JournalofDevelopmentEffectiveness1:195-216.Kluve,J.(2010)."TheEffectivenessofEuropeanActiveLaborMarketPrograms."LabourEconomics17:904-918.Kroft,Kory,FabianLange,MatthewJ.Notowidigdo,andLawrenceF.Katz(2016)."Long-termUnemploymentandtheGreatRecession:TheRoleofComposition,DurationDependence,andNon-Participation".JournalofLaborEconomics34(S1):S7-S54.Krueger,AlanB.,JuddCramer,andDavidCho(2014)."AretheLong-TermUnemployedontheMarginsoftheLaborMarket?"BrookingsPapersonEconomicsActivitySpring:229-280.Lalonde,RobertJ(1986).“EvaluatingtheEconometricEvaluationsofTrainingProgramswithExperimentalData.”AmericanEconomicReview76(4):604-620.Lalonde,RobertJ(2003)."EmploymentandTrainingPrograms."InRobertA.Moffit,editor,MeansTestedTransferProgramsintheUnitedStates.Chicago:UniversityofChicagoPress.Lechner,MichaelandConnieWunsch(2009).“AreTrainingProgramsMoreEffectivewhenUnemploymentisHigh?”JournalofLaborEconomics27(4):653-692.

36

Martin,JohnP.(2014)"ActivationandActiveLabourMarketPoliciesinOECDCountries:StylizedFactsandEvidenceonTheirEffectiveness."IZAPolicyPaperNo.84.Meyer,BruceD.(1995)"LessonsfromtheU.S.UnemploymentInsuranceExperiments."JournalofEconomicLiterature33(1):91-131.Mincer,Jacob.(1974).Schooling,ExperienceandEarnings.NewYork:NationalBureauofEconomicResearch.Roberts,ColinD.andT.D.Stanley,editors(2005).Meta-RegressionAnalysis:IssuesofPublicationBiasinEconomics.NewYork:Wiley.Rothstein,HannahR.,AlexanderJ.SuttonandMichaelBorenstein(2005).PublicationBiasinMetaAnalaysis-Prevention,AssessmentandAdjustments.NewYork:Wiley.Schochet,PeterZ.,JohnBurghardt,andStevenGlazerman(2001).NationalJobCorpsStudy:TheImpactsofJobCorpsonParticipants’EmploymentandRelatedOutcomes.Princeton,NJ:MathematicaPolicyResearch.Sianesi,B.(2004).AnEvaluationoftheSwedishSystemofActiveLaborMarketProgramsinthe1990s."ReviewofEconomicsandStatistics86(1):133-155.Smith,JeffreyandPetraTodd(2005).“DoesMatchingOvercomeLaLonde’sCritiqueofNonexperimentalMethods?”JournalofEconometrics125(1-2):305-353. Stanley,T.D.(2008)."Meta-RegressionMethodsforDetectingandEstimatingEmpiricalEffectsinthePresenceofPublicationSelection."OxfordBulletinofEconomicsandStatistics70(1):102-127.StanleyT.D.andHristosDoucouliagos(2012).Meta-RegressionAnalysisinEconomicsandBusiness.MiltonParkUK:Routledge.Solon,Gary,StevenJ.HaiderandJeffreyM.Wooldridge(2015)."WhatAreWeWeightingFor?"JournalofHumanResources50(2):301-316.Sutton,A.J.,S.J.Duval,R.L.Tweedie,K.R.AbramsandD.R.Jones(2000).“EmpiricalAssessmentofEffectofPublicationBiasonMeta-analyses”.BritishMedicalJournal320:1574–77.UnitedStatesDepartmentofLabor(2016).LaborForceStatisticsfromtheCurrentPopulationSurvey2015,http://www.bls.gov/cps.

37

Wolfson,PaulJ.andBelman,Dale(2015.)"15YearsofResearchonU.S.EmploymentandtheMinimumWage".TuckSchoolofBusinessWorkingPaperNo.2705499.

Figure 1: Number of Program Estimates, By Year of Program Start

0

10

20

30

40

50

60

70

80

90

100

1980 1989 1994 1999 2004 2009

Year of Program Start

Num

ber o

f Program

Estim

ates

Experimental Design

Non‐experimental Design

Figure 2a: Short Term Effects and Confidence Intervals

-0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4Estimated Short Term Effect

Note: Diamonds represent estimated short term treatment effects on probability of employment for a program/participant subgroup (PPS). Horizontal lines represent 95% confidence intervals. Graph shows 56 estimates -- 2 large positive estimates are not shown for clarity.

precision-weighted mean = 0.028unweighted mean = 0.053(median = 0.029)

Figure 2b: Medium Term Effects and Confidence Intervals

-0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4Estimated Medium Term Effect

Note: Diamonds represent estimated medium term treatment effects on probability of employment for a program/participant subgroup (PPS). Horizontal lines represent 95% confidence intervals. Graph shows 69 estimates -- 3 large positive estimates are not shown for clarity.

precision-weighted mean = 0.053unweighted mean = 0.086 (median = 0.051)

Figure 2c: Long Term Effects and Confidence Intervals

-0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4Estimated Long Term Effect

Note: Diamonds represent estimated long term treatment effects on probability of employment for a program/participant subgroup (PPS). Horizontal lines represent 95% confidence intervals. Graph shows 39 estimates.

precision-weighted mean = 0.109unweighted mean = 0.123(median = 0.090)

0

2000

4000

6000

8000

10000

12000

14000

-0.1 -0.05 0 0.05 0.1 0.15 0.2 0.25

Precision

ofE

stim

ate

EstimatedShortTermEffect

Figure3a:FunnelPlotofShortTermEstimates

Note:Diamondsrepresentprecisionofestimatedshorttermtreatmenteffectforaprogram/participant(PPS)subgroup,graphed againsttheestimatedtreatmenteffect.Graphshows56estimates-- 2largepositiveestimatesarenotshownforclarity.

unweightedmeantreatmenteffect=0.053

t=2boundaries

precision-weightedmean=0.028

0

2000

4000

6000

8000

10000

12000

14000

-0.15 -0.1 -0.05 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35

Precision

ofE

stim

ate

EstimatedMediumTermEffect

Figure3b:FunnelPlotofMediumTermEstimates

Note:Diamondsrepresentprecisionofestimatedmediumtermtreatmenteffectforaprogram/participantsubgroup(PPS),graphedagainsttheestimatedtreatmenteffect.Graphshows69estimates-- 3largepositiveestimatesarenotshownforclarity.

meantreatmenteffect=0.086

t=2boundaries


0

2000

4000

6000

8000

10000

12000

14000

-0.1 -0.05 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

Precision

ofE

stim

ate

EstimatedLongTermEffect

Figure3c:FunnelPlotofLongTermEstimates

Note:Diamondsrepresentprecisionofestimatedlongtermtreatmenteffectforaprogram/participant(PPS)subgroup,graphedagainsttheestimatedtreatmenteffect.Graphshows39estimates.

meantreatmenteffect=0.123

t=2boundaries


-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

5.0

6.0

-5

0

5

10

15

20

25

30

35

40

1985 1987 1989 1991 1993 1995 1997 1999 2001 2003 2005 2007

GDP

Gro

wth

Rate

Dur

ing

Year

of P

rogr

am

Mea

n Pr

ogra

m E

ffect

for P

rogr

ams

in a

Yea

r

Figure 4: Mean Program Effects and GDP Growth Rate - German ALMP Evaluations

Mean Program Effect, Left Scale GDP Growth Rate, Right Scale

Note: mean program effect on the employment rate of ALMP participants (in percentage points) is plotted for programs offered in different years, with number of program estimates in that year. GDP growth rate is for the first year the program was operated.

721 1

4

27

5

17

6

3

6

Table 1: Description of Sample of Program Estimates

Austria, U.S., U.K, Latin Amer.

Germany, Nordic Aust., N.Z., and

Full sample Switzerland Countries Canada Non‐OECD Carribean

(1) (2) (3) (4) (5) (6)

Number of estimates 857 290 212 87 132 72Number of PPS's 526 163 127 45 86 54Number of studies 207 52 48 24 33 19

Type of program (%):

Training 49 62 17 45 79 97

Job Search Assistance 15 8 26 22 2 0

Private Subsidy 14 17 15 5 11 3

Public Employment 9 9 10 3 6 0

Other 14 5 32 25 2 0

Age of program group (%):

Mixed 59 54 61 72 40 25

Youth (<25 years) 21 12 20 15 53 69

Older (≥25 years) 20 33 19 13 8 6

Gender of Program group (%):

Mixed 54 53 67 43 43 11

Males only 22 24 18 25 23 44

Females only 23 23 16 32 31 44

Type of Program Participants (%):

Registered unemployed 65 86 67 33 24 0

Long‐term unemployed 12 8 10 25 7 0

Disadvantaged 23 6 23 41 69 100

Outcome of Interest (%):

Employment status 57 83 31 26 63 54

Earnings 21 8 25 47 36 43

Hazard to new job 12 7 25 3 0 0

Other hazard 6 0 16 2 0 3

Unemployment status 4 2 4 21 1 0

Effect Measured at (%):

Short Term 48 42 54 37 47 57

Medium Term 35 34 31 40 45 42

Long Term 16 23 16 23 8 1

Experimental Design (%) 19 0 39 31 28 26

Note: see text for description of sample. Study refers to an article or unpublished paper. PPS refers to a

program/participant subgroup (e.g., a job search assistance program for mixed gender youths). Estimate refers

to an estimate of the effect of the program on the participant subgroup at either a short‐term (<1 year after

completion of the program), medium term (1‐2 years post completion) or long term (2+ years post completion)

time horizon. A9Job search assistance programs include sanction programs. "Other" programs include those

that combine elements of the four distinct types.

Country Group:

Table2:SummaryofProgramEstimatesbyAvailabilityofEstimateofProgramEffectonProbabilityofEmployment

Subsamplewith SubsamplewithOutcome= EstimateofEffect MeanEffect MedianEffect

Fullsample Prob.ofEmp. onProb.ofEmp. (std.error)a (std.error)a

(1) (2) (3) (4) (5)

Numberofestimates 857 490 352

NumberofPPS's 526 274 200

Numberofstudies 207 111 83

ShortTermEstimates

AllSTestimates--number[pct.oftotalestimates] 415[48] 205[42] 141[40] 1.6(0.8) 1.0(1.0)

SignificantpositiveSTestimate--pct.ofSTsample 40 31 33 8.8(1.3) 6.0(1.3)

InsignificantSTestimate--pct.ofSTsample 42 47 44 0.5(0.4) 0.0(0.6)

SignificantnegativeSTestimate--pct.ofSTsample 18 22 23 -6.4(0.8) -5.0(0.7)

MediumTerm(MT)Estimates

AllMTestimates--number[pct.oftotalestimates] 301[35] 194[40] 143[41] 5.4(1.2) 3.0(0.7)

SignificantpositiveMTestimate--pct.ofMTsample 52 50 47 11.3(1.9) 8.5(1.1)

InsignificantMTestimate--pct.ofMTsample 40 41 43 1.3(0.3) 1.0(0.4)

SignificantnegativeMTestimate--pct.ofMTsample 8 9 10 -5.0(1.2) -4.9(2.2)

LongTerm(LT)Estimates

AllLTestimates--number[pct.oftotalestimates] 141[16] 91[19] 68[19] 8.7(2.2) 4.9(1.4)

SignificantpositiveLTestimate--pct.ofsample 61 65 65 13.0(2.7) 9.0(2.2)

InsignificantLTestimate--pct.ofsample 35 32 32 1.3(0.6) 1.1(0.7)

SignificantnegativeLTestimate--pct.ofsample 4 3 3 -4.2(0.5) --

aStandarderrorsareclusteredbystudy.

SignandSignificanceofProgramEffects: EstimatedProgramEffectonProb.OfEmp(col.3subsample)

352

200

83

Notes:seenotetoTable1.Shorttermprogramestimatesarefortheperiodupto1yearafterthecompletionoftheprogram.Mediumtermestimatesarefortheperiodfrom1to2yearsaftercompletionoftheprogram.Longtermestimatesarefortheperiod2ormoreyearsaftercompletionoftheprogram.Effectsizesareonlyavailableforstudiesthatmodeltheprobabilityofemploymentastheoutcomeofinterest,andprovideinformationonmeanemploymentrateofcomparisongroup.

Table 3a: Comparison of Impact Estimates by Program Type and Participant Group

MedianNumber Sample Percent Short Medium Longer Short Medium Longer

Est's. Size RCT's Term Term Term Term Term Term(1) (2) (3) (4) (5) (6) (7) (8) (9)

All 857 10,709 19.4 1.6 5.4 8.7 40 52 61(141) (143) (68) (415) (301) (141)

By Program Type:Training 418 7,700 12.9 2.0 6.6 6.7 35 54 67

(90) (92) (35) (201) (163) (54)

Job Search Assist. 129 4,648 51.2 1.2 2.0 1.1 53 63 43(16) (13) (7) (68) (40) (21)

Private Subsidy 118 10,000 8.5 1.1 6.2 21.1 37 65 88(13) (17) (16) (49) (37) (32)

Public Sector Emp. 76 17,084 0.0 3.6 -1.1 0.8 32 25 27(14) (12) (6) (41) (24) (11)

Other 116 17,391 31.0 7.2 5.8 2.0 52 38 43(8) (9) (4) (56) (37) (23)

By Intake Group:UI Recipients 554 11,000 17.1 -0.1 4.3 8.5 34 47 59

(93) (101) (50) (258) (193) (103)

Long Term Unem. 106 8,900 16.0 5.8 13.0 12.7 50 65 63(17) (16) (10) (50) (40) (16)

Disadvantaged 197 7,027 27.4 4.2 5.3 5.0 50 59 68(31) (26) (8) (107) (68) (22)

Mean Program Effect on Prob. Emp. (×100) Pct. Of Estimates with Sig. Positive Impact

Notes: see Tables 1 and 2. Number of program estimates associated with each table entry is reported in parentheses. Program effects (columns 4-6) are only available for studies that model the probability of employment as the outcome of interest.

Table 3b: Additional Comparisons of Impact Estimates by Participant Groups and Design

MedianNumber Sample Percent Short Medium Longer Short Medium Longer

Est's. Size RCT's Term Term Term Term Term Term(1) (2) (3) (4) (5) (6) (7) (8) (9)

All 857 10,709 19.4 1.6 5.4 8.7 40 52 61(141) (143) (68) (415) (301) (141)

By Age:Mixed Age 505 10,000 16.6 1.7 6.7 10.5 47 57 65

(71) (84) (51) (238) (178) (89)

Youth (<25) 180 3,000 33.3 2.9 2.7 0.2 32 41 67(34) (29) (5) (92) (64) (24)

Non-Youth 172 25,850 12.8 0.1 4.5 4.6 31 51 43(36) (30) (12) (85) (59) (28)

By Gender:Mixed Gender 466 11,000 19.7 1.6 4.4 5.6 39 52 59

(89) (85) (45) (224) (155) (87)

Males Only 191 10,000 15.2 1.4 6.1 13.1 41 50 58(24) (28) (9) (95) (72) (24)

Females Only 200 8,345 22.5 4.1 7.8 15.9 41 55 70(28) (30) (14) (96) (74) (30)

By Evaluation Design:Experimental 166 1,471 100.0 4.4 2.5 0.5 40 41 37

(28) (25) (15) (78) (58) (30)

Non-experimental 691 16,000 0.0 0.9 6.0 11.0 40 55 68(113) (118) (53) (337) (243) (111)

Mean Program Effect on Prob. Emp. (×100) Pct. Of Estimates with Sig. Positive Impact

Notes: see Tables 1 and 2. Number of program estimates associated with each table entry is reported in parentheses. Program effects (columns 4-6) are only available for studies that model the probability of employment as the outcome of interest.

Table 4: Transitions in Program Impacts for a Given Program and Participant Subgroup

(1) (2) (3) (4) (5) (6)All 0.021 0.024 -0.006 0.231 0.250 0.020

(0.008) (0.015) (0.004) (0.055) (0.103) (0.052) Number Studies 105 43 47 225 100 102

By Program TypeTraining 0.032 0.044 -0.004 0.314 0.439 0.048

(0.008) (0.018) (0.005) (0.072) (0.085) (0.049) Number Studies 70 28 28 121 41 42

Job Search Assist. 0.003 -0.001 0.000 0.265 0.143 -0.111(0.008) (0.001) (0.002) (0.095) (0.167) (0.144)

Number Studies 10 7 7 34 21 18

Private Subsidy -0.020 -0.004 -0.012 0.083 0.167 -0.062(0.049) (0.078) (0.013) (0.150) (0.267) (0.068)


Public Sector Emp. 0.016 -0.049 -0.019 0.158 -0.143 -0.143(0.014) (0.049) (0.019) (0.170) (0.494) (0.285)


Sanction/Threat 0.004 -0.021 -0.014 0.000 0.158 0.211(0.012) (0.008) (0.006) (0.108) (0.182) (0.092)

Number Studies 6 4 4 27 19 19Notes: Change in estimated effect size in column 1 represents the difference between the estimated medium term and short term effects on the probability of employment for a given program and participant subgroup (PPS). Changes in columns 2 and 3 are defined analogously. Change in sign/significance in column 4 is defined as +1 if the short term estimate is significantly negative and the medium term estimate is insignificant, or if the short term estimate is insignificant and the medium term estimate is significantly positive; 0 if the sign and significance of the short term and medium term estimates is the same; and -1 if the short term estimate is significantly positive and the medium term estimate is insignificant, or if the short term estimate is insignificant and the medium term estimate is significantly negative. Changes in columns 5 and 6 are defined analogously. Standard deviations (clustered by study number) in parenthesis.

Change in Program Effect on Prob. Of Emp. Change in Sign/Significanceshort term to medium term

short term to long term

medium term to long term

short term to medium term

short term to long term

medium term to long term

Table 5: Estimated Program Effects on Probability of Employment

(1) (2) (3) (4) (5)Effect Term (Omitted = Short Term)Medium Term 0.035 0.029 0.045 0.040 0.032

(0.012) (0.009) (0.016) (0.011) (0.008)Long Term 0.064 0.045 0.046 0.045 0.035

(0.019) (0.015) (0.018) (0.017) (0.014)Program Type (Omitted = Training)Job search Assist. -0.032 -0.009 -0.008 0.007 --

(0.012) (0.020) (0.011) (0.021)Private Subsidy 0.042 0.042 -0.009 0.016 --

(0.030) (0.026) (0.037) (0.041)Public Sector Emp. -0.065 -0.08 -0.056 -0.058 --

(0.013) (0.020) (0.014) (0.023)Other 0.007 0.003 0.052 0.047 --

(0.027) (0.036) (0.026) (0.038)

Interaction with Medium Term:Job search Assist. -- -- -0.037 -0.037 -0.028

(0.019) (0.018) (0.011)Private Subsidy -- -- 0.005 -0.012 -0.048

(0.045) (0.044) (0.046)Public Sector Emp. -- -- -0.02 -0.024 -0.015

(0.027) (0.027) (0.015)Other -- -- -0.059 -0.048 -0.028

(0.022) (0.021) (0.014)

Interaction with Long Term :Job search Assist. -- -- -0.048 -0.03 -0.034

(0.019) (0.022) (0.014)Private Subsidy -- -- 0.153 0.088 -0.061

(0.060) (0.056) (0.044)Public Sector Emp. -- -- -0.002 -0.053 -0.061

(0.028) (0.039) (0.031)Other -- -- -0.098 -0.109 -0.051

(0.030) (0.037) (0.015)

Additional Controls No Yes No Yes

Notes: Sample size is 352 estimates. Standard errors (clustered by study) in parentheses. Models are linear regressions with the effect size as dependent variable. Coefficients of additional control variables are reported in Table 7. Model in column 5 is estimated with fixed effects controlling for 200 program participant subgroups.

PPS fixed effects

Table 6: Models for Sign/Significance of Estimated Program Effects

(1) (2) (3) (4) (5) (6) (7)

Effect Term (Omitted = Short Term)

Medium Term 0.372 0.483 0.563 0.639 0.491 2.008 0.387

(0.088) (0.099) (0.130) (0.138) (0.145) (0.452) (0.158)

Long Term 0.597 0.742 0.901 1.053 1.03 2.536 0.457

(0.157) (0.167) (0.175) (0.171) (0.206) (0.449) (0.135)

Program Type (Omitted = Training)

Job search Assist. 0.274 0.286 0.531 0.532 0.569 ‐‐ ‐‐

(0.156) (0.168) (0.180) (0.197) (0.459)

Private Subsidy 0.139 0.076 ‐0.04 ‐0.132 ‐0.166 ‐‐ ‐‐

(0.189) (0.210) (0.224) (0.263) (0.438)

Public Sector Emp. ‐0.677 ‐0.758 ‐0.383 ‐0.489 ‐1.399 ‐‐ ‐‐

(0.219) (0.228) (0.276) (0.279) (0.496)

Other ‐0.11 ‐0.205 0.318 0.202 1.148 ‐‐ ‐‐

(0.172) (0.184) (0.206) (0.236) (0.653)

Interaction with Medium Term:

Job search Assist. ‐‐ ‐‐ ‐0.289 ‐0.283 ‐0.004 ‐0.743 ‐0.157

(0.235) (0.249) (0.343) (0.618) (0.218)

Private Subsidy ‐‐ ‐‐ 0.138 0.226 0.353 ‐1.085 ‐0.266

(0.289) (0.311) (0.486) (1.025) (0.289)

Public Sector Emp. ‐‐ ‐‐ ‐0.645 ‐0.573 0.051 ‐1.394 ‐0.229

(0.285) (0.288) (0.477) (0.861) (0.305)

Other ‐‐ ‐‐ ‐0.764 ‐0.705 ‐0.662 ‐2.04 ‐0.387

(0.226) (0.245) (0.278) (0.796) (0.234)

Interaction with Long Term :

Job search Assist. ‐‐ ‐‐ ‐1.017 ‐1.022 ‐0.832 ‐1.842 ‐0.331

(0.313) (0.294) (0.313) (0.775) (0.258)

Private Subsidy ‐‐ ‐‐ 0.611 0.58 1.274 ‐2.295 ‐0.385

(0.375) (0.387) (0.798) (1.428) (0.350)

Public Sector Emp. ‐‐ ‐‐ ‐0.643 ‐0.675 0.131 ‐2.366 ‐0.45

(0.490) (0.497) (0.832) (1.568) (0.822)

Other ‐‐ ‐‐ ‐0.999 ‐1.021 ‐1.638 ‐0.735 ‐0.194

(0.353) (0.375) (0.430) (1.282) (0.344)

Additional Controls No Yes No Yes Yes PPS fixed

effects

PPS fixed

effects

Notes: Sample size is 857 program estimates, except column 5, which is based on 352 estimates for which

program effect on probability of employment is also available. Standard errors (clustered by study) in

parentheses. Models in columns 1‐6 are ordered probits, fit to ordinal data with value of +1 for significantly

positive, 0 for insignificant, ‐1 for significantly negative estimate. Estimated cutpoints (2 per model) are not

reported in the Table. Coefficients of additional control variables are reported in Table 7. Model in column 6 is

estimated with fixed effects controlling for program participant subgroups. Model in column 7 is a linear

regression with fixed effects for program participant subgroups.

Table 7: Estimated Coefficients of Control Variables Included in Models in Tables 5 and 6

(1) (2) (3) (4) (5)Outcome of Interest (Omitted = Probability of Employment)Earnings -- -- -0.003 -0.01 --

(0.130) (0.132)Hazard to New Job -- -- 0.275 0.264 --

(0.211) (0.212)Other Hazard -- -- 0.613 0.547 --

(0.275) (0.263)Unemployment Status -- -- 0.598 0.591 --

(0.293) (0.285)Age of Program Group (Omitted = Mixed)Youths (<25) -0.031 -0.025 -0.368 -0.348 -0.518

(0.018) (0.018) (0.151) (0.153) (0.287)Older (>=25) -0.07 -0.063 -0.423 -0.425 -0.671

(0.020) (0.020) (0.157) (0.160) (0.297)Gender of Program Group (Omitted = Mixed)Males only 0.011 0.007 -0.007 -0.006 -0.328

(0.022) (0.022) (0.149) (0.149) (0.266)Females only 0.047 0.041 0.064 0.053 0.000

(0.023) (0.023) (0.144) (0.146) (0.250)Country Group (Omitted = Nordic)Germanic 0.056 0.045 0.250 0.176 0.910

(0.027) (0.026) (0.192) (0.196) (0.488)Anglo -0.026 -0.028 0.177 0.14 1.231

(0.030) (0.028) (0.241) (0.236) (0.579)East Europe 0.022 0.028 0.131 0.096 0.618

(0.031) (0.028) (0.201) (0.202) (0.378)Rest of Europe 0.016 0.012 0.125 0.088 0.738

(0.024) (0.023) (0.187) (0.189) (0.483)Latin America 0.009 0.012 0.108 0.1 1.012

(0.053) (0.053) (0.338) (0.338) (0.826)Remaining Countries 0.035 0.038 -0.063 -0.064 1.124

(0.035) (0.035) (0.281) (0.286) (0.529)Type of Program Participant (Omitted = Registered Unemployed)Disadvantaged 0.018 0.013 0.542 0.527 0.356

(0.036) (0.036) (0.228) (0.228) (0.623)Long-term Unemployed 0.083 0.08 0.388 0.404 0.392

(0.032) (0.031) (0.181) (0.179) (0.332)

-0.029 -0.023 -0.135 -0.122 -0.55(0.016) (0.016) (0.179) (0.177) (0.232)

Randomized Experimental -0.009 -0.008 -0.065 -0.095 -0.314 Design (0.020) (0.019) (0.170) (0.170) (0.332)

Square Root of Sample Size -0.003 0.001 0.159 0.098 0.484(0.042) (0.037) (0.184) (0.191) (0.706)

Published Article -0.024 -0.026 -0.203 -0.213 -0.41(0.017) (0.017) (0.133) (0.132) (0.254)

Citations Rank Index -0.001 -0.001 0.007 0.005 -0.005(0.002) (0.001) (0.012) (0.012) (0.024)

R-squared/ Log Likelihood 0.36 0.40 -765 -752 -283Notes: Standard errors (clustered by study) in parentheses. Coefficients in columns 1 and 2 are from models reported in columns 2 and 4 of Table 5. Coefficients in columns 3-5 are from models reported in columns 2, 4, and 5 of Table 6. See notes to Tables 5 and 6 for more information.

Program Effect - OLS Models Specifications in Table 5 (cols 2,4)

Sign/Significance - Ordered Probit Models Specifications in Table 6 (cols 2,4,5)

Program Duration - Dummy if > 9 months

Table 8: Tests for Publication Bias (Funnel Asymmetry Tests)

Control for Time

Horizon Only

Basic Controls (col.

2 of Table 5)

Interacted Controls

(col. 4 of Table 5)

PPS Fixed Effects (col.

5 of Table 5)

(1) (2) (3) (4)

Estimated coefficient of sampling error of estimated program effect:

Unweighted OLS 0.007 0.009 0.009 0.001

(0.005) (0.008) (0.008) (0.014)

Precision‐Weighted 0.022 0.035 0.032 ‐0.005

(0.031) (0.024) (0.020) (0.019)

Notes: Entry in each row and column corresponds to estimated coefficient of sampling error of estimated

program effect from a different specification. Models in column 1 include only a constant and dummies for

medium term and long term time horizon as additional controls; models in columns 2, 3 and 4 include same

controls included in specifications in columns 2, 4 and 5 of Table 5, respectively. Unweighted models are fit by

OLS. Precision‐weighted models are fit by weighted least squares using Winsorized inverse sampling variance

of estimated program effect as weight. Weight is Winsorized at 10th and 90th percentiles, respectively,

corresponding to values of 19.2 and 10000, respectively. Standard errors, clustered by study, in parentheses.

Table 9: Comparison of Relative Impacts of Different Program Types on Different Participant Groups

All Job Search Private Sector Public SectorProgram Types Training Assistance Job/Subsidy Employment Other

(1) (2) (3) (4) (5) (6)

Number Estimates 352 217 36 46 32 21Number of Studies 83 51 15 19 14 8Mean Effect Size (×100) 4.52 4.71 1.48 9.91 -1.83 5.65

Constant 0.024 0.020 -0.003 -0.053 -0.004 0.106(0.015) (0.014) (0.017) (0.043) (0.027) (0.022)

Medium Term 0.032 0.041 0.017 0.028 0.020 0.004(0.009) (0.010) (0.014) (0.045) (0.018) (0.019)

Long Term 0.054 0.055 0.005 0.133 0.009 -0.011(0.016) (0.016) (0.012) (0.068) (0.023) (0.014)

Intake Group (Base=Regular UI Recipients)Disadvantaged -0.002 -0.020 0.037 0.053 (omitted) 0.030

(0.019) (0.020) (0.017) (0.026) (0.030)Long Term Unemployment 0.072 0.122 0.024 0.088 0.029 -0.109

(0.032) (0.059) (0.018) (0.038) (0.030) (0.019)Gender Group (Base=Mixed)Male 0.019 0.028 (omitted) 0.110 -0.048 -0.022

(0.020) (0.025) (0.069) (0.031) (0.023)Female 0.054 0.058 (omitted) 0.163 -0.024 -0.111

(0.022) (0.028) (0.050) (0.030) (0.025)Age Group (Base=Mixed)Youth -0.037 -0.030 0.009 0.034 -0.059 (omitted)

0.018 0.021 0.011 0.037 0.030Older Participants -0.048 -0.059 0.016 -0.094 -0.047 0.046

(0.019) (0.024) (0.021) (0.053) (0.038) (0.002)

Controls for Program Typea Yes No No No No No

aFour dummies for different types of programs included.

Notes: standard errors, clustered by study, in parenthesis. See note to Table 5.

Table 10: Impacts of Macro Conditions on the Effectiveness of ALMP's

Baseline +Country Effects +GDP Growth Baseline +GDP Growth +Unemp. Rate

(1) (2) (3) (4) (5) (6)

Medium Term 0.032 0.030 0.028 0.043 0.034 0.040

(0.009) (0.009) (0.009) (0.009) (0.008) (0.009)

Long Term 0.054 0.046 0.040 0.056 0.031 0.048

(0.016) (0.017) (0.015) (0.020) (0.014) (0.020)

GDP Growth Rate (%) ‐‐ ‐‐ ‐0.010 ‐‐ ‐0.032 0.034

(Unemp. Rate in col. 6) (0.006) (0.008) (0.011)

Program Type (Base=Training)

Job Search Assistance ‐0.033 ‐0.053 ‐0.056 ‐0.076 ‐0.103 0.010

(0.017) (0.028) (0.027) (0.034) (0.023) (0.069)

Private Sector Job/Subsidy 0.031 0.027 0.019 0.020 ‐0.002 0.012

(0.026) (0.028) (0.028) (0.029) (0.024) (0.031)

Public Sector Employment ‐0.079 ‐0.074 ‐0.069 ‐0.096 ‐0.073 ‐0.102

(0.019) (0.027) (0.025) (0.029) (0.024) (0.025)

Other Programs ‐0.015 ‐0.045 ‐0.066 ‐0.096 ‐0.188 ‐0.104

(0.033) (0.027) (0.033) (0.034) (0.046) (0.050)

Intake Group (Base=Regular UI Recipients)

Disadvantaged ‐0.002 0.000 0.010 0.046 0.106 0.050

(0.019) (0.034) (0.033) (0.028) (0.035) (0.027)

Long Term Unemployed 0.072 0.098 0.095 0.112 0.109 0.092

(0.032) (0.033) (0.031) (0.034) (0.027) (0.033)

Gender Group (Base=Mixed)

Female 0.019 0.048 0.053 0.066 0.081 0.052

(0.020) (0.025) (0.024) (0.026) (0.022) (0.022)

Male 0.054 0.086 0.092 0.094 0.111 0.081

(0.022) (0.030) (0.030) (0.033) (0.030) (0.029)

Age Group (Base=Mixed)

Youth ‐0.037 ‐0.026 ‐0.026 ‐0.024 ‐0.057 ‐0.040

(0.018) (0.018) (0.020) (0.023) (0.020) (0.051)

Older Participants ‐0.048 ‐0.055 ‐0.062 ‐0.073 ‐0.097 ‐0.066

(0.019) (0.024) (0.025) (0.028) (0.023) (0.023)

Country Dummies No Yes Yes Yes Yes Yes

All Available Program Effect Estimates Denmark, France, Germany, and US Only

Notes: standard errors, clustered by study, in parenthesis. Models in columns 1‐3 are fit to 352 program estimates from 83 studies, with mean of dependent

variable = 0.0452. Models in columns 4‐5 are fit to 200 program estimates from Denmark, France, Germany, and the U.S. from 38 studies, with mean of

dependent variable = 0.0423. Model in columns 6 is fit to 181 program estimates from the same four countries from 34 studies, with mean of dependent variable

= 0.0441.

Appendix Figure 1: Distribution of Program Estimates by Country

224

258

313241

1158

26

42253

11212443

192659

125

424

12388

213

84

166

6612

813

57

0 25 50 75 100 125 150 175 200 225 250 275

ArgentinaAustraliaAustriaBelgium

BrazilBulgariaCanadaChina

ColombiaCzech

DenmarkDominican

EstoniaFinlandFrance

GermanyHungary

IndiaIrelandIsraelItaly

JordanKoreaLatvia

MalawiMexico

NetherlandsNZ

NicaraguaNorwayPanama

PeruPoland

PortugalRomania

RussiaSerbia

SlovakiaSlovenia

South AfricaSpain

Sri LankaSweden

SwitzerlandTurkey

UKUS

Number of Program Estimates

0

5

10

15

20

25

30

-0.12 -0.08 -0.04 0 0.04 0.08 0.12 0.16 0.2 0.24 0.28 0.32 0.36 0.4

Num

bero

fEstim

ates

RangeofEstimatedProgramEffect:MidpointofInterval

AppendixFigure2a:HistogramofShortTermEstimatedEffects

SignificantlyNegative

Insignificant

SignificantlyPositive

0

5

10

15

20

25

30

-0.12 -0.08 -0.04 0 0.04 0.08 0.12 0.16 0.2 0.24 0.28 0.32 0.36 0.4

Num

bero

fEstim

ates


AppendixFigure2b:HistogramofMediumTermEstimatedEffects


Insignificant


0

2

4

6

8

10

12

-0.12 -0.08 -0.04 0 0.04 0.08 0.12 0.16 0.2 0.24 0.28 0.32 0.36 0.4

Num

bero

fEstim

ates


AppendixFigure2c:HistogramofLongTermEstimatedEffects


Insignificant


050

100

150

num

ber o

f est

imat

es

1985-1989 1990-1994 1995-1999 2000-2004 2005-2008Time of Program Operation

Training JSA Private Public

Appendix Figure 3a: Number of estimates by program type over time

0.2

.4.6

perc

enta

ge


positive insignificant negative

Sign/Significance Short Term

0.2

.4.6

perc

enta

ge



Sign/Significance Medium Term0

.2.4

.6.8

perc

enta

ge



Sign/Significance Long Term

0.0

5.1

.15

effe

ct s

ize


short run medium run long run

Program Effect

Appendix Figure 3b: Program Effects over Time

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Under-0.075

-0.075to-0.05

-0.05to-0.025

-0.025to0

0to0.025

0.025to0.05

0.05to0.075

0.075to0.10

0.10to0.125

0.125to0.15

0.15to0.175

0.175to0.20

0.20to0.225

0.225to0.25

0.25to0.275

0.275to0.30

0.30to0.325

0.325to0.35

0.35to0.375

Over0.375

Fractio

n

RangeofEstimatedProgramEffect

AppendixFigure4:EffectDistributionsConditionalonSign/SignificanceFemalesvs.OtherParticipantGroups

Males/MixedGender- Sign.Negative Females- Sign.NegativeMale/MixedGender- Insignficant Females- InsignificantMales/MixedGender- Sign.Positive Females- Sign.Positive

Note:figureplotshistogramsforestimatedprogrameffectsonprobabilityofemployment,classifyingparticipantgroupsaseitherfemaleormale/mixedgender,andclassifyingestimatedprogrameffectsasnegativeandsignificant,insignificant,orpositiveandsignificant.Thereareatotalof352estimatedprogrameffects:72forfemalesand280formales/mixedgender.

Appendix Table 1a: Transitions between Sign‐ and Significance Categories, All Estimates

Short‐term Estimates Significantly Positive Insignificant Significantly Negative

Significantly Positive (N=80) 84 16 0

Insignificant (N=95) 37 60 3

Significantly Negative (N=50) 20 46 34





Medium‐term Estimates Significantly Positive Insignificant Significantly Negative




Note: see notes to Table 2.









Medium‐term Estimates Significantly Positive Insignificant Significantly Negative




Note: see notes to Table 2.

Percent of Long‐term Estimates


Appendix Table 1b: Transitions between Sign‐ and Significance Categories, Subsample with

Program Effects

Percent of Medium‐term Estimates

Percent of Medium‐term Estimates



Appendix Table 2a: Models for Standardized Effect Size

(1) (2) (3) (4)


Medium Term 0.071 0.056 0.101 0.088

(0.027) (0.021) (0.037) (0.025)

Long Term 0.131 0.091 0.097 0.099

(0.044) (0.038) (0.040) (0.040)


Job search Assist. ‐0.059 ‐0.012 0.002 0.029

(0.027) (0.043) (0.026) (0.044)

Private Subsidy 0.094 0.086 ‐0.007 0.044

(0.068) (0.057) (0.091) (0.099)

Public Sector Emp. ‐0.120 ‐0.152 ‐0.081 ‐0.084

(0.034) (0.044) (0.055) (0.062)

Other 0.036 0.007 0.139 0.108

(0.071) (0.094) (0.068) (0.098)


Job search Assist. ‐0.098 ‐0.092

(0.043) (0.041)

Private Subsidy ‐0.016 ‐0.055

(0.102) (0.104)

Public Sector Emp. ‐0.081 ‐0.09

(0.070) (0.073)

Other ‐0.133 ‐0.105

(0.048) (0.045)



(0.041) (0.052)

Private Subsidy 0.329 0.182

(0.142) (0.127)


(0.081) (0.108)

Other ‐0.239 ‐0.273

(0.073) (0.092)


Number of Observations 352 352 352 352

R Squared 0.13 0.33 0.21 0.37

Notes: Standard errors (clustered by study) in parentheses. Models are linear

regressions with the standardized effect size (program effect on probability of

employment, divided by standard deviation of employment rate of comparison

group) as dependent variable. Additional regressors see Table 7.

Appendix Table 2b: Models for Proportional Program Effects

(1) (2) (3) (4)


Medium Term 0.062 0.037 0.134 0.111

(0.057) (0.054) (0.059) (0.037)

Long Term 0.120 0.053 0.103 0.117

(0.083) (0.079) (0.051) (0.044)


Job search Assist. ‐0.102 ‐0.086 ‐0.023 ‐0.049

(0.043) (0.090) (0.053) (0.096)

Private Subsidy 0.157 0.09 0.019 0.075

(0.119) (0.100) (0.185) (0.202)

Public Sector Emp. ‐0.021 ‐0.113 0.204 0.17

(0.159) (0.116) (0.379) (0.350)

Other 0.100 ‐0.073 0.265 0.069

(0.172) (0.213) (0.171) (0.218)



(0.073) (0.066)

Private Subsidy ‐0.031 ‐0.133

(0.194) (0.198)


(0.381) (0.399)

Other ‐0.19 ‐0.123

(0.080) (0.077)



(0.063) (0.082)

Private Subsidy 0.45 0.16

(0.291) (0.240)


(0.410) (0.513)

Other ‐0.437 ‐0.497

(0.171) (0.188)



R Squared 0.04 0.23 0.09 0.28

Notes: Standard errors (clustered by study) in parentheses. Models are linear

regressions with the proportional effect size (program effect on probability of

employment, divided by mean employment rate of comparison group) as dependent

variable. Additional regressors see Table 7.

Appendix Table 3: Alternative Models for Subsample with Estimated Program Effect

(1) (2) (3) (4)

Medium Term 0.029 0.489 0.429 ‐0.719

(0.009) (0.114) (0.137) (0.173)

Long Term 0.045 0.969 0.851 ‐1.365

(0.015) (0.221) (0.218) (0.437)


Job search Assist. ‐0.009 0.443 0.507 ‐0.531

(0.020) (0.436) (0.434) (0.528)

Private Subsidy 0.042 0.252 0.733 0.475

(0.026) (0.300) (0.314) (0.315)

Public Sector Emp. ‐0.08 ‐1.356 ‐1.239 1.349

(0.020) (0.287) (0.330) (0.335)

Other 0.003 0.55 0.987 0.595

(0.036) (0.542) (0.492) (0.628)

Age of Program Group (Omitted = Mixed)

Youths (<25) ‐0.031 ‐0.614 ‐0.643 1.052

(0.018) (0.282) (0.322) (0.499)

Older (>=25) ‐0.07 ‐0.735 ‐0.657 0.845

(0.020) (0.280) (0.272) (0.428)

Gender of Program Group (Omitted = Mixed)

Males only 0.011 ‐0.289 ‐0.591 ‐0.101

(0.022) (0.273) (0.310) (0.279)

Females only 0.047 0.043 ‐0.196 ‐0.476

(0.023) (0.251) (0.272) (0.315)

Country Group (Omitted = Nordic)

Germanic 0.056 1.033 1.043 ‐1.351

(0.027) (0.460) (0.498) (0.454)

Anglo ‐0.026 1.265 1.312 ‐‐

(0.030) (0.577) (0.625)

East Europe 0.022 0.615 0.644 ‐0.579

(0.031) (0.358) (0.447) (0.437)

Rest of Europe 0.016 0.825 0.909 ‐0.984

(0.024) (0.469) (0.560) (0.456)

Latin America 0.009 1.017 1.334 ‐1.075

(0.053) (0.826) (0.879) (1.170)

Remaining Countries 0.035 1.161 1.138 ‐‐

(0.035) (0.521) (0.618)

Type of Program Participant (Omitted = Registered Unemployed)

Disadvantaged 0.018 0.428 0.294 ‐0.99

(0.036) (0.618) (0.588) (1.022)

Long‐term Unemployed 0.083 0.436 0.481 ‐0.512

(0.032) (0.311) (0.325) (0.332)

Program Duration > 9 Months ‐0.029 ‐0.599 ‐0.526 0.563

(0.016) (0.234) (0.265) (0.326)

Experiment ‐0.009 ‐0.312 ‐0.677 ‐0.95

(0.020) (0.330) (0.395) (0.461)

Square Root of Samplesize ‐0.003 0.471 0.796 0.817

(0.042) (0.689) (0.851) (0.719)

Published Article ‐0.024 ‐0.374 ‐0.41 0.328

(0.017) (0.252) (0.277) (0.298)

Citations Rank Index ‐0.001 ‐0.009 0.003 0.057

(0.002) (0.023) (0.024) (0.034)


R Squared/ Log Likelihood 0.356 ‐288 ‐190 ‐92

Notes: Standard errors (clustered by study) in parentheses. Dependent variables are: estimated program effect in column

1; ‐1, 0 or 1 indicating sign/significance in column 2; indicator for significant positive effect in column 3; indicator for

significant negative effect in column 4. Sample in column 4 excludes observations that are perfectly predicted.

Probit for Sig.

Positive

Probit for Sig.

Negative

Ordered Probit for

Sign/Significance

OLS Model for

Program Effect

Appendix Table 4: Re‐normalized Program Effects on Probability of Employment

(1) (2)

Program Type = Training

Short Term 0.020 0.010

(0.009) (0.010)

Medium Term 0.066 0.053

(0.017) (0.014)

Long Term 0.067 0.058

(0.018) (0.018)

Program Type=Job Search Assistance


(0.010) (0.019)


(0.008) (0.017)

Long Term 0.011 0.037

(0.016) (0.021)

Program Type=Private Subsidy


(0.037) (0.038)


(0.027) (0.032)

Long Term 0.211 0.192

(0.043) (0.038)

Program Type=Public Sector Employment

Short Term ‐0.036 ‐0.045

(0.013) (0.012)

Medium Term ‐0.011 ‐0.020

(0.011) (0.021)

Long Term 0.008 ‐0.028

(0.016) (0.029)

Program Type=Other


(0.024) (0.039)


(0.022) (0.038)

Long Term 0.020 0.015

(0.028) (0.050)

Additional Controls No Yes

Notes: Sample size is 352 estimates. Standard errors (clustered by study) in

parentheses. Models correspond to specifications estimated in columns 3 and 4 of

Table 5, but are re‐normalized to give estimated impacts by program type and time

horizon (at mean values of all covariates for model in column 2).

AppendixTable5:ImpactsofMacroConditionsontheEffectivenessofALMP's

(1) (2) (3) (4) (5) (6) (7) (8)

IndicatorforMediumTermImpact 0.028 0.032 0.032 0.036 0.034 0.041 0.040 0.039(0.009) (0.009) (0.011) (0.011) (0.008) (0.008) (0.009) (0.009)

IndicatorforLongTermImpact 0.040 0.035 0.046 0.049 0.031 0.035 0.048 0.050(0.015) (0.013) (0.018) (0.018) (0.014) (0.013) (0.020) (0.019)

MeasuresofMacroconditions:

Conditionsoverprogramperiod -0.010 0.006 -0.032 0.034(0.006) (0.007) (0.008) (0.011)

Conditionsinfirstyearofprogram -0.006 0.012 -0.007 0.022(0.003) (0.007) (0.004) (0.013)

Conditionsinyearafterprogramend 0.013 -0.003 0.020 0.011(0.006) (0.006) (0.006) (0.011)

NumberofObservations 352 351 333 314 200 200 181 175

CountryDummies Yes Yes Yes Yes Yes Yes Yes YesNotes:standarderrors,clusteredbystudy,inparenthesis.Macroconditionsaremeasuredasaveragesovertheyearsofprogramimplementationincolumns(1),(3),(5),(7).Conditionsaremeasuredinthefirstyearofprogramimplementationandtheyearaftertheendofprogramimplementationincolumns(2),(4),(6),(8).AllmodelsincludecountryfixedeffectsandthesamesetofcovariatesasinTable10.

AllAvailableProgramEffectEstimates: EffectsforDenmark,France,Germany,andUSOnly:

GDPGrowth UnemploymentRate GDPGrowth UnemploymentRate

Date post:	27-Jun-2020
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

What Works? A Meta Analysis of Recent Active Labor Market Program Evaluations · 2017-04-05 ·...

Documents