Funding What You Want To Sustain

May 282:00 – 3:30pm EST

Today’s Agenda

2

• Overview of Sustainability Series

• Return on Investment: Kevin Hollenbeck, Upjohn Institute

• Braided Funding: Nate Anderson, Jobs for the Future

• Closing out the Sustainability Series Major takeaways Advice for rounds 1 & 2

Framework for Sustainability

• Key elements Deciding what to sustain Coordinating ongoing work External activities Funding

Assessing and reducing costs Obtaining new funding sources

3

4

Presenter

Return on InvestmentBenefit-Cost Analysis of WIF Initiatives

Kevin HollenbeckVice President, Senior Economist

W.E. Upjohn Institute for Employment Researchhollenbeck@upjohn.org

5

Outline

• Stages/Types of Program Evaluation• Basics• Components of a Benefit-Cost Analysis• Five Ingredients for ROI Study and Example• Concluding Thoughts

6

Stages/Types of Program Evaluation

I. Process Evaluation (Also called Implementation or Formative)

Purpose is to document and assess the fidelity of the start-up and ongoing processes to design/proposal.

Audience: Program administratorsMay be developmental (evaluator makes suggestions)

7

Stages/Types of Program Evaluation (2)

II. Net Impact Evaluation (Also called Outcome or Summative)

Purpose is to measure or estimate the outcomes that result from the program or intervention relative to a counterfactual situation, which is usually no program (or status quo).

Audience: Program fundersAttributing outcomes to the program is often difficult. RCTs are usually considered most reliable way to do this. Other approaches include regression discontinuity or statistical matching.

8

Stages/Types of Program Evaluation (3)

III. Benefit-Cost Analysis/Return on Investment (ROI)

Purpose is to determine if the value of the outcomes exceeds the cost

Audience: Program funders/taxpayers

If B/C > 1.0, then we conclude that the program was worth pursuing. If a decision-maker has multiple options for a program, it is most rational to pursue the option with the highest B/C.

If ROI > going interest rate, then we conclude that the program was worth pursuing. Decision-makers should invest in programs with highest ROI.

9

Poll #1

What type(s) of evaluation activities are you undertaking for your WIF grants? (Check all that apply.)

• Process (aka implementation or formative)?• Net impact (aka outcome or summative)?• Benefit-cost or ROI?

10

Poll #2

How familiar are you with the notion of discounting an investment?

• Very familiar• Somewhat familiar• Not at all familiar

11

Basics

Investments

When you make a financial investment, you commit funds today (e.g., buy an asset) and you get returns in the future.

12

Basics (2)

We use a complicated present value formula to determine the value of your investment because future dollars (when you get the return) are worth less than today’s dollars (when you make the investment)

The present value formula involves discounting with the interest rate.

13

Basics (3)

WIF initiative is an investment:

You are spending resources today with the expectation of receiving benefits in the future, (e.g., more efficient job searches, better outcomes for customers, etc.)

14

Basics (4)

Just like a financial investment, you need to discount the future benefits. The number that you use for discounting is like an interest rate, but it is arbitrary.

Discount rates are usually in the range of 0.00 to 0.10, like an interest rate.

Because they are in the denominator, a higher discount rate will lower the PV of the WIF investment. (Note: we often say that youth make poor decisions because they have very high discount rates – they don’t value highly the future.)

15

Basics (5)

Benefit-Cost Ratio:

PV(WIF) or PV(I) = Benefit-Cost Ratio WIF I

• Rational to invest/offer services if Benefit-Cost Ratio > 1, which means NPV > 0

• Usually express as $1 invested today will return (benefit-cost ratio – 1) dollars in the future

16

Basics (6)

Return on Investment (ROI): ROI = [ PV(WIF) – WIF ] / WIF

= Benefit-Cost ratio - 1

Reported as:• percentage terms (multiply by 100)• payback period (careful about period of analysis)• invest $1 and get back a net gain of $ROI• annual rate of return

17

Basics (7)

Example: Each dollar spent in a program returns $6.50 in 15 years.

Note: This statement is ambiguous about whether this is a benefit-cost ratio of 6.50 in which case the ROI is 5.50, or if this is an ROI of 6.50, in which case the benefit-cost ratio is 7.50. Assume that is an ROI of 6.50. Then, • This program has a return of 650%. • This program has a payback period of 15/6.50 = 2.31

years (27.69 months). • This program has an annual return of 13.29 percent.

18

Components of a Benefit-Cost Analysis

Costs

• Usually easier to measure than benefits

• Desired concept is marginal full costs; marginal means relative to the counterfactual; full means actual “out of pocket” plus opportunity costs

• WIF grant will generally be actual cost

19

Components of a Benefit-Cost Analysis (2)

Costs

• Opportunity costs – value of “in kind” contributions, including volunteer time; also if participants are investing more time than in status quo, need to value their participation time.

• On the cost side, it is best (most conservative) to overestimate costs.

20

Poll #3

As you think about your WIF initiative, will you be incurring costs beyond the WIF grant from U.S. DOL (e.g. opportunity costs of participants, cash or in kind contributions)?

o Yes

o No

21

Components of a Benefit-Cost Analysis (3)

BenefitsDeciding what to include and how to measure the benefits, (B1, B2, . . ., Bt) is usually very difficult. Usually constrained to monetary benefits only.

On the benefit side, it is best (most conservative) to underestimate benefits.

Monetary Benefits Nonmonetary BenefitsEarnings HealthTransfer income Knowledge and skillsTaxes paid Confidence/self-esteemUnemployment compensation Family stability

Noncognitive skills

22

Components of a Benefit-Cost Analysis (4)

Cost Effectiveness Analysis

If it is very difficult/impossible to monetize benefits, then you may do a cost effectiveness analysis. This analysis examines multiple ways to accomplish an outcome, and identifies the method that minimizes costs.

Example: Medical office use of transparency for consent form (10,000 patients)

Hard copy and scan: paper, filing time $600Transparency and scan: transparency, pen, erase time $120

23

Components of a Benefit-Cost Analysis (5)

Multiple Stakeholders (have different ROIs):• Participants• Employers• Public (taxpayers)• Society (benefits and costs derived by adding

across participants + employers + taxpayers)

ROI to society (typically monetary benefits) not the same as social ROI (tries to monetize non-monetary benefits)

24

Components of a Benefit-Cost Analysis (6)

Benefit or Cost Participants

EmployersRest of Society All

WIF Initiative Costs 0 0/- - -

Participant Costs - 0 0 -

Productivity of individuals who become employed

0 + + +

Higher earnings + - 0/+ 0/+

Fringe benefits + - 0 0/+

Less unemp./turnover

- + + +

Less income transfers

- 0 + 0

Higher taxes - 0/- + 0

Net benefits + -/+ -/0/+ -/0/+

25

Five Ingredients for ROI Study and Example

I. Identification of “treatment” and treated populations

II. Counterfactual; Identification of benefits and causality

III. Time period of analysesIV. CostsV. Treatment of statistical uncertainty

26

Five Ingredients for ROI Study and Example (2)

I. Identification of treatment and treated populations

entrance or exit cohort

entire population of participants or subpopulation (by office, by characteristic, etc.)

27

Five Ingredients for ROI Study and Example (3)

II. Counterfactual; Identification of benefits and causality

Just because an event preceded an outcome doesn’t mean that the event caused the outcome.

In other words, we can’t assume that services provided to a customer result in the customers’ outcomes.

28

Five Ingredients for ROI Study and Example (4)

Counterfactual: circumstances that would occur if investment was not made (action was not taken)

Example: Did ARRA work or not work?Need to find a counterfactual

29

Five Ingredients for ROI Study and Example (5)

III. Time period of analyses

Countervailing tendencies: Policy makers want results sooner rather than later

Outcomes are often better after considerable time has past

Advice: Try to obtain data that have outcomes at least 3 years after service delivery; try to avoid doing lifetime extrapolations (More art than science).

30

Five Ingredients for ROI Study and Example (6)

IV. Costs As noted, want marginal full costs; usually easy to access

V. Treatment of statistical uncertainty Important to communicate

31

Five Ingredients for ROI Study and Example (7)

$ in 2010; estimated with data from Washington state; comparison group is applicants who were not served (program participants deleted); discount rate is 0.03; average age at exit = 39.4

Results First 2.5 Years Lifetime (age 65)

Earnings increment $9,034 $52,812

Fringe benefits 1,806 10,562Taxes -1,559 - 9,110

Unemployment comp. 410 1,072

TANF/FS/Medicaid - 892 - 2,158

Forgone earnings - 707 - 707Program cost 9,347 9,347

32

Five Ingredients for ROI Study and Example (7)

Results First 2.5 Years Lifetime

Individual -- --

Taxpayers -54.40% 0.34%

Society (annual) 2.69% 6.32%

$1 invested earns $1.24 $6.86

33

Five Ingredients for ROI Study and Example (8)

I. Treatment: Exiters in 2005/06 and 2007/08II. Identification Strategy: Regression analysis using

eligible clients who did not get served as non-treatment population

III. Time period: short-term: + 3 quarters; long-term: + (9 -12) quarters

IV. Cost: per participant cost supplied by state agencyV. Statistical uncertainty: t-statistic from regression

analysis

34

Concluding Thoughts

1. ROI is a tautological construct, just likeNet worth = Assets – Liabilities

Consequently, difference in “studies” is only the difference in how the key ingredients are addressed.

35

Concluding Thoughts (2)

2. Can/should ROIs be used for cross-program accountability?

In theory, a state legislature/governor should want to invest marginal resources in programs where the ROI is highest. Problem is that different programs have different administrative sets (and reliability). Furthermore assumptions used may be quite different.

My advice: at best, ROI is one number that should be used along with lots of other evidence.

36

Concluding Thoughts (3)

3. Can these tools be used for program improvement, or are they just useful for accountability?

Jury is out. I believe that they can/should be used for program improvement by comparing different subpopulations/service modalities/etc. But I have not seen such applications.

37

Concluding Thoughts (4)

4. Always “annualize” results.

$1 returning $5.42 may be better or worse than $1 returning $6.71. It all depends on the time period.

Consequently for consistency and comparative analyses, I believe that ROIs should always report annual ROI in addition to other metrics.

38

Concluding Thoughts (5)

5. Quality of ROI analysis hinges greatly on the quality of the data

GIGO.

39

Questions?

40

Presenter

Nate AndersonDirector, Jobs for the Future

41

Braided Funding – 3 Components

Funding Streams

Strategies

Services and Support

42

Benefits to Braided Funding

•To the organization: Improves coordination and alignment

among partners Reduces duplication and realizes

efficiencies Expands capacity in resource-short

environments Brings additional staff strengths and

expertise to customers Increases flexibility in service planning Improves program impacts and

outcomes Supports scalability, sustainability, and

new investment

•To the customer: Improves system accessibility,

transparency, and ease of use Offers more creative/diverse

service delivery options Brings additional staff

expertise and assistance to customer needs

Improves customer experience and value

Improves customer outcomes/results

43

WIOA and Braided Funding

• Multi-program, unified state strategic planning• Increased emphasis on Board role in aligning and coordinating

workforce programs• Collaborative planning and service strategies

on a regional level• Enhanced flexibility in funding and service design• Integrated intake, assessment, and service delivery• Elimination of “sequence of service”• Partner co-location and dedicated funding from

Career Center partners for shared costs• Support for “pay for performance”• Aligned performance indicators across expanded “core”

programs

44

Significant Resources: Federal

•Job Seekers & Other Costs: WIA Adult, Dislocated Worker,

and Youth Temporary Assistance for

Needy Families Wagner-Peyser Supplemental Nutrition

Assistance Program Employment and Training

Trade Adjustment Assistance Adult Education and Family

Literacy

Reemployment Eligibility Assessment and Services

Veterans’ Services/Benefits

Community Services, Community Development, and Social Services Block Grants

Vocational Rehabilitation Carl D. Perkins Pell Grants, other aid

45

Significant Resources: Other

• State Funding (scholarships, economic development, grant funds, etc.)

• Employer/industry contributions• Community- and faith-based, non-profit• Foundations/philanthropic• Fee-for-service• Private grants

46

The Importance of Partnerships

•Service delivery level: At the local/regional level, engage all

service providers in ongoing partnership efforts to build and coordinate support for comprehensive service delivery and resource integration

•Policy Level: Develop, formalize, and sustain interagency

partnerships among organizations that fund and oversee services for job seeker and employer customers

47

Steps to Braiding Funds

1. Braided Funding for What? Focus on a single model, population or

collaborative structure. Why do you think braided funding is important?

2. Identifying Funding Priorities Break down your model/population/structure

by need, impact and cost. Where do you have the most need? Prioritize which pieces you want to fund, and in

what order.

48

Steps to Braiding Funds

3. List Funding Possibilities Brainstorm about funding sources for each high-

priority goal. Identify questions you need answered, and who

might answer them. Reach out!4. Develop a Pitch

Partnerships to braid funds must benefit everyone.

Integrate evidence. Customize to the customer.

49

Steps to Braiding Funds

5. Develop a Work Plan Set goals (including interim). Identify leads for each piece of the plan. Include timeline.

6. Go Go Go! It is okay to move slowly. Keep focused, and keep the effort moving. Pay attention to new funding sources that may

emerge.

50

The Reality of Braided Funding

• Braided funding must benefit all partners, but not always financially.

• Braided funding works best if you have a narrow and specific goal in mind.

• Some organizations are better positioned to receive funding, while others are better positioned to give funding (investing).

• Start small to demonstrate the concept, iron out the wrinkles.

• This is a long-term strategy.

51

The Workforce System and Braided Funding

• Must be integrated into a partnership• Consider cost-savings (efficiency)

Case Management Co-location Service Delivery (e.g. online, call centers)

• Revenue Fee for Service, Discretionary Grants

• Data Management• Flexibility within workforce funding sources?

52

Questions?

WIF Sustainability Series Retrospective

Thank you for participating in the series!

Recorded events and resources available here:

•http://innovation.workforce3one.org/grantees/resources-view/from-innovating-to-sustaining

53

Poll Question

54

•Which aspect of sustainability planning requires the most attention?

Deciding what to sustain Coordinating ongoing work Communications Scaling and replicating Funding

55

Poll Question

• What is one highlight or takeaway from either this session or the series as a whole?

Open-ended response

56

Highlights from the Series

• Deciding What to Sustain In using data…

Collect as much data as possible Focus on what you can collect rather than might be ideal

but inaccessible In selecting project elements to sustain…

Start broad and keep focusing in as you plan Find project elements that can be used in multiple

contexts Strategic planning should guide actions, funding comes

last

57

Highlights from the Series

• Coordinating Ongoing Work Decide who will lead the partnership Engage high-level leadership, and proactively

engage new leaders when partner organizations experience turnover

Think about how your innovations can benefit a wider range of partners

Balance the success of the grant with success of the larger vision: Know when to push and when to be responsive

58

Highlights from the Series

• Getting the Word Out When deciding how and what to scale…

Be realistic and realize that not everything can be scaled Determine what elements can be flexible and change Communication is core to scaling

Help your message stand out from the growing flood of information… Be as specific as possible in your goals and strategy Keep your audience in mind – your message is about

them and not about your organization Describe success that matters to others

59

Poll Question

• For Round 1 Grantees only – what is one piece of advice you would give Round 2 Grantees about planning for sustainability?

Open-ended response

60

Parting Words of Wisdom

• For Round 1 Grantees:

Remember that sustainability is an iterative process – revisit your sustainability plan regularly!

• For Round 2 Grantees

Start early!