www.impaqint.com JANUARY 2016 Evaluating & Enhancing Public Programs & Policies 1

SCOTT DAVIS, PHD

ISSUE BRIEF

Current Efforts to Make Education and Training Program Scorecards Available to American Workers

In recent years, the Federal government has supported the wider use of data and reliable evidence in shaping policy, including finding ways that data can be made available to the public to help people make informed decisions. For example, CollegeScorecard.ed.gov is a college scorecard website hosted by the U.S. Department of Education (DOE) that allows users to compare costs, graduation rates, and the post-college salaries of graduates for colleges and universities across the country.

Scorecard from collegescorecard.ed.gov

The goal of making this information easily and publicly accessible is to help students make more informed college choices. Just as prospective college students must decide which colleges are best for them, American workers looking to upgrade their skills to compete for jobs—or who are interested in gaining new skills to advance their careers—must choose from among a number of education or training programs. Without reliable information, making these choices can be difficult. One of the core missions of the U.S. Department of Labor (DOL) is to help American workers achieve these objectives by supporting a workforce system that helps workers develop skills to succeed in the labor market and that helps connect unemployed or underemployed workers to jobs.

The U.S. invests a significant amount of money in helping American workers enhance or increase their own skills. Because education and training programs are not all equally effective in leading to good jobs, individuals need reliable information they can use to help make good choices about which programs are best suited for their job and career goals. To help consumers make better decisions about which training options to pursue, a number of states have developed state education and training program scorecards. This issue brief outlines the need for and goals of such scorecards, key success factors in developing scorecards, and the future of education and training scorecards.

Access more IMPAQ issue briefs at www.impaqint.com/briefs

www.impaqint.com JANUARY 2016 Evaluating & Enhancing Public Programs & Policies 2

ISSUE BRIEF Current Efforts to Make Education and Training Program Scorecards Available to American Workers

Without reliable information, making these choices can be difficult. One of the core missions of the U.S. Department of Labor (DOL) is to help American workers achieve these objectives by supporting a workforce system that helps workers develop skills to succeed in the labor market and that helps connect unemployed or underemployed workers to jobs.

Much of the employment and training support provided through DOL is authorized by the Workforce Innovation and Opportunity Act (WIOA). A key feature of WIOA is the idea of consumer choice—individuals that receive funding to enroll in training programs are meant to decide for themselves which programs will best meet their needs.

Despite the minimum quality standards included in WIOA, not all training programs are equally worthwhile. Moreover, there is evidence to suggest that some people have difficulty comparing training alternatives and, as a result, they can select programs in fields where there may be limited job opportunities, where positions pay low wages, or in which they may struggle to complete the training.i

To help consumers make better decisions about which training options to pursue, a number of states have developed state education and training program scorecards.

Much like the DOE college scorecard website, state scorecards generally allow users to compare key features of different programs, such as cost and duration. Importantly, state scorecards also report on the labor market outcomes of program participants, such as the proportion of program graduates that were employed post-completion, or the average earnings of program completers.

State of Washington Scorecard, from careerbridge.wa.gov.

In spite of the potential value of scorecards for helping trainees make good choices, only a relatively small number of states have well-developed scorecard systems. To better understand how state scorecards work and the factors associated with their success, in 2013, DOL contracted with IMPAQ to analyze scorecards among states that had been awarded grants under the first three rounds of DOL’s Workforce Data Quality Initiative (WDQI).ii In addition to finding that (at the time of our report) few states had successfully developed scorecard systems, the study also concluded that many states would likely find it difficult to do so.

EIGHT KEY FACTORS FOR SUCCESSFUL SCORECARDS Among the 29 states that had received WDQI funding, IMPAQ found that only five—Florida, Minnesota, New Jersey, Virginia, and Washington—had successfully developed scorecards that were based on reliable data.iii So that DOL could understand both what had supported the successful development of scorecards in these states and the potential for other states to develop scorecards, IMPAQ analyzed the development of each of the five scorecard systems. Based on conversations with state officials and a thorough

www.impaqint.com JANUARY 2016 Evaluating & Enhancing Public Programs & Policies 3

ISSUE BRIEF Current Efforts to Make Education and Training Program Scorecards Available to American Workers

document review, IMPAQ identified eight key factors associated with the successful implementation of state scorecards among WDQI states:

1) State-level administration of the system,2) Use of individual-level data to measure

outcomes,3) Having the necessary data infrastructure,4) Successfully joining education and workforce

data, 5) Having the analytical capacity to work with the

data,6) Having a favorable interpretation of the Family

Educational Rights and Privacy Act,7) Successfully obtaining participation from

education and training providers, and8) Having a broad and steady institutional

commitment.iv

Our analysis of the five successful WDQI states suggested that states that lack these eight factors are not likely to be able to develop or sustain scorecards.

THE FUTURE OF EDUCATION AND TRAINING PROGRAM SCORECARDS After the 2013 study was successfully completed, DOL contracted with IMPAQ to conduct another scorecards study. The focus of the second study was to determine the feasibility of implementing a national approach to developing state scorecards. The basic idea behind a national approach would be for DOL to gather data from states on participants in education and training programs, facilitate matching that data to earnings data from national databases (such as from income tax records), construct relevant scorecard measures based on the matched data, and then make the results available to the public.

The two key potential benefits of this approach are:

1) It could provide a mechanism for states that donot have the internal capacity to developscorecards, and

2) It would overcome some of the limitationsassociated with using state-level earnings datato calculate program outcome measures.v

The second study is underway and will be competed in the summer of 2016. IMPAQ is working with three states—Ohio, Missouri, and New Jersey—to obtain individual-level data on participants in WIOA-funded education and training programs, along with state earnings data. The participant data will be matched to earnings data from the National Directory of New Hires (NDNH), which is comprised of earnings data from all states. Using the matched data, IMPAQ will analyze the differences in typical labor market outcome measures used in scorecards, comparing measures based on single-state UI wage record data to measures based on the NDNH. This study will also document the process of working with states to obtain the data and match it to a federal source of earnings information for all states, identifying challenges to this approach and comparing it to the state model. Ultimately, our work will help policymakers better understand critical issues involved in supporting efforts to make scorecards more widely available.

CONCLUSION The U.S. invests a significant amount of money in supporting a workforce system that aims to help strengthen the nation’s workforce by helping American workers invest in their own skills. Because education and training programs are not all equally effective in leading to good jobs, individuals need reliable information they can use to make good choices about which programs are best suited for their job and career goals.

www.impaqint.com JANUARY 2016 Evaluating & Enhancing Public Programs & Policies 4

ISSUE BRIEF Current Efforts to Make Education and Training Program Scorecards Available to American Workers

Better understanding both the potential for a federally-supported system of state scorecards and how scorecard outcome measures are influenced by the source of earnings data will help identify promising alternatives for the future development of these systems.

At a time when policymakers face pressure to make efficient use of taxpayer dollars, providing better information to potential trainees through reliable scorecards can help ensure that money dedicated to funding education and training for American workers is spent wisely.

ACKNOWLEDGEMENTS Dr. Scott Davis, Senior Research Associate

Dr. Davis (Ph.D., Economics, University of Virginia) is a labor economist with expertise in workforce programs, economic modeling, analytical methods, and public policy analysis.

He has nine years of research and evaluation experience, including eight years of project management experience. At IMPAQ, he has been

responsible for designing and implementing research projects incorporating quasi-experimental methods, implementation studies, surveys, and rigorous statistical analyses.

IMPAQ International IMPAQ International helps governments, businesses, foundations, non-profits, and universities evaluate and enhance their programs and policies.

With more than 30 researchers, including Ph.D. level economists, psychologists, sociologists, and social scientists, as well as experts with advanced degrees in public policy, our team brings a thorough knowledge of the workforce investment system.

Through interdisciplinary teams, we rigorously evaluate which policies and interventions are most effective, informing our clients how best to improve a program, scale up an approach that works, or modify an intervention that does not.

Call (443) 259-5500 or visit IMPAQint.com to learn how we can help evaluate and enhance your programs and policies.

www.impaqint.com JANUARY 2016 Evaluating & Enhancing Public Programs & Policies 5

ISSUE BRIEF Current Efforts to Make Education and Training Program Scorecards Available to American Workers

REFERENCES

1 See, for example, Louis S. Jacobson and Robert J. LaLonde, Using Data to Improve the Performance of Workforce Training, Washington, DC: Brookings Institution (April 2013). Available online at http://bit.ly/1KSeBCw.

2 The states that had received WDQI grants through the first three rounds of funding were Arkansas, Florida, Hawaii, Idaho, Illinois, Iowa, Kansas, Louisiana, Maine, Maryland, Massachusetts, Michigan, Minnesota, Missouri, Nebraska, New Jersey, North Carolina, North Dakota, Ohio, Oklahoma, Oregon, Pennsylvania, Rhode Island, South Carolina, South Dakota, Texas, Virginia, Washington, and Wyoming.

3 Two other states, Louisiana and Texas, were in the final stages of developing and releasing scorecard systems at the time of our writing.

4 Our full report is available online at http://1.usa.gov/1Dx9bMN.

5 Among the state scorecard systems our original report identified, all relied on single-state Unemployment Insurance (UI) wage record data to identify earnings- and employment-related outcomes for program participants. One advantage of using UI wage record data is the reliability of the earnings amounts that are reported, compared to other possible data sources such as participant follow-up surveys. Two disadvantages are that UI wage records do not cover all types of employment and that they only include earnings from jobs within a single state. Although the Wage Record Interchange System (WRIS) and WRIS2 allow states to share UI wage record data, restrictions on the use of the data generally preclude using it for the purposes of creating or maintaining education and training program scorecards.