1

DHHS and Outcome Measurement

Wes Albinger, Contract Services Coordinator

Milwaukee County

Department of Health and Human Services

Contract Administration

March 16th, 2006

Presentation will be available at:

http://county.milwaukeecounty.org/ContractMgt15483.htm

Starting Monday, March 20, 2006

2

Intended “Outcomes” of OM Initiative

Initial: Agencies understand DHHS definition and application of OM concepts. Agencies understand the direction DHHS is headed with Outcome Measurement and their role in that process.

Intermediate: DHHS articulates long term outcomes for each program area. Agency program activities and initial outcomes are aligned with DHHS long term Outcomes identified for each program.

Long Term: Indicators and data collection protocols are established for each program area. OM is

part of program and contract requirements.

3

DHHS and Outcomes: WHY?

To establish an objective standard which is consistently applied to measure and report program impact.

To provide DHHS and Agencies with a meaningful way to report success to stakeholders (e.g. client, provider, funding agencies, etc.).

o To clearly communicate expectations (goals and objectives) to provider agencies.

4

DHHS and Outcomes: WHY?

To promote client choice.

To increase benefits for clients from contract dollars.

To improve program performance via feedback and program improvement.

5

Purpose of Today’s presentation

This is a multi-year project that will begin simply with establishment of outcomes and a requirement that agencies report to us how they think they are achieving those outcomes. At this point, DHHS will not be holding programs to a specific set of indicators. In subsequent years, contracts may actually include not only the outcomes, but also the indicators and performance goals.

None of this is etched in stone yet -- we have called you together at this juncture specifically to gather your input.

6

FAQQ: Is DHHS telling us how to run our program?

A: No. Outcome Measurement focuses more on the impact of the program than the program itself. While the outcome is affected by the program, we are more concerned with the “end” of the program than the “means” to that end. Basically, we are interested in defining where we want programs to go, but not how to get there.

In some instances, external factors prescribe program outcomes or processes. An example would be the new Federal Birth to Three Outcomes which will be required for all BT3 programs. Additionally, the growth in popularity of evidence based program design is putting increasing pressure on programs to adopt proven approaches to service delivery. These will likely play an increasing role in programs.

7

FAQQ: What role will Agencies play in this process?

o A: DHHS will articulate long term outcomes for each program, distilled from estimation of client need, agency practice and capacity, current program requirements, federal, state and other mandates, internal staff, and research. DHHS will solicit information from agencies relating to outcomes, indicators, and data collection methods to help inform this process. Keep in mind that on a program by program basis, the amount of discretionary “slack” in setting outcomes varies greatly.

o Throughout the process, DHHS will release information for review and comment.

8

FAQ Q: What are the implications of these changes for my Agency in the short term?

A: For 2007, we will not be prescribing indicators and performance goals in association with the long-term outcomes. Long term outcomes will be identified for several pilot programs for the FY 2007 RFP. For these programs, you will be asked to describe in detail how your program design and outcomes relate to the long term outcome identified for your program, using successful past performance or a research based justification. Additionally, you will be asked to identify what indicators you are or propose to use to measure the achievement of the outcome, as well as your data collection and or measurement tool(s) (concepts explained shortly).

9

FAQ, implications for short term, cont’d

For those programs without DHHS identified long term outcomes in the FY 2007 RFP, you will be asked to describe your current or proposed program design and outcomes, using successful past performance or an evidence based justification. Additionally, you will be asked to identify what indicators you are or propose to use to measure the achievement of the outcome, as well as your data collection and or measurement tool(s).

10

FAQ, implications for short term, cont’d

In other words, tell DHHS:

What do you intend to do? (program design)

Why do you intend to do it? How does what you are proposing contribute toward the long term outcome? (rationale/justification)

In what way(s) do you expect clients to benefit? (short, intermediate, and long-term outcomes)

How will you know if you have accomplished this? (indicators)

11

FAQ

o Q: What are the implications of these changes for my Agency in the long term?

o A: Indicators and data collection protocols for each program area will be developed through an inclusive process. OM will be part of program and contract requirements.

12

FAQ

o Q: How will this initiative create work for my agency?

o A: This initiative is more focused on reporting differently, versus reporting additional information. The required reporting will be part of your semi annual evaluation reports.

o DHHS will seek to minimize the administrative burden. One way this will be accomplished is by soliciting Agency input to determine current capacity and potential technical assistance needs.

13

Example-Agency input

It is worth noting that DHHS has done an extensive review of existing outcome data currently collected to help formulate our thinking and strategy. Many of the examples used in this presentation are actual outcome data already collected; we are looking at the content currently being provided as a starting point.

14

Agency Input

You may recall recently completing an Outcome Measurement Survey. These were sent to all agencies with purchase of service contracts. Some notable results, which have already been used to inform our progress on this initiative, are as follows:

15

Survey responses

Agencies have widely interpreted definitions of “Outcome”; as such, there are widely varied applications of Outcome Measurement.

16

Survey responses

o Most indicated an interest in an outcome measurement presentation.o Most indicated an interest in a formatted spreadsheet for measuring outcomes. o Vast majority are currently reporting outcomes.o Approximately half cited fundor requirements as the basis for deciding to report outcomes. It appears that many agencies are not finding an internal benefit to

tracking outcomes, but are doing it because they “have” to.

17

Survey responses

o Most already track outcomes for ALL programs.o Most common theme that emerged as “concern” was administrative/cost burden, with the second being data reliability and validity.o Vast majority have developed a logic model or other visual representation of main programs.o Most common methodology employed for measuring outcomes was United Way’s model.

18

Technical Assistance

DHHS will provide technical assistance throughout this process.

Let DHHS know what your technical assistance needs are.

19

On to Outcomes

This is intended to give you a sense of the process DHHS is using internally to determine outcomes, and may be helpful to you as you identify how your agency’s program will work toward a specific long term outcome.

DHHS will be using a methodology which is already familiar to many of you, and enjoys a broad base of support. It has become most familiar to practitioners through the United Way, but it is used more widely than that.

20

Some thoughts on

Terminology

Throughout this presentation, I will be using specific terminology. The concept behind the terminology is more important; however, because there is a wide use of terminology and interpretation of terminology among agencies, it is important that:

We adopt a common language for discussing Outcome Measurement concepts, and that

We use it consistently.

21

What are Outcomes and Why Measure Them?

OUTCOMES-

The intended benefit(s) for participants during or after their involvement with a program. You can also think of Outcomes as the OBJECTIVE of the program.

22

Assumptions and Organizational Challenges to implementing OM

Support for OM is based on the assumption that the information produced creates value that exceeds the time and effort to collect it.

The challenge, then, is to make sure that the information that OM produces is useful, and that collecting it is administratively as simple as possible.

DHHS wants OM information to serve useful purposes for all affected stakeholders.

23

Assumptions and Organizational Challenges to implementing OM

Concern about OM also comes from the belief that the outcomes measured are not an accurate or complete reflection of the program.

o Our challenge is to identify outcomes which are appropriate. Your input will help

24

An Outcomes “Makeover”

The Before

Served 350 people. Hosted 10 resume writing seminars. Provided 150 hours of individual career counseling. 25 staff attended Outcome Measurement training.

Notice emphasis on delivery of service, or the program itself.

25

An Outcomes “Makeover”

Gee, you sure did a lot, but what happened???

Did your clients get jobs?

Did your clients learn anything?

Did your clients change their lifestyle?

In general, how were your clients affected or changed?

26

An Outcomes “Makeover”The After

70% (160 out of 230) of clients over 50 years of age obtained employment .

Of the total, 65% were able to maintain employment for at least 1 year.

Notice emphasis on the benefit or change to the person served. No (or little) emphasis on the actual program. In fact, the program can only be inferred from the outcome statement. These statements reflect the OBJECTIVE of the program.

27

What Exactly is An Outcome?

Benefits for participants during or after their involvement with a program.

Change in knowledge or skills

Change in

behavior

Change in condition

28

Examples of Outcomes

Clients decreased emergency room visits.

Participants significantly reduced alcohol/drug use.

Immigrants were able to secure legal residency and citizenship.

Staff demonstrate proficiency with Outcome Measurement concepts.

29

Outcome or not?

Clients increase community activities.

Clients participate in skill training programs.

Residents feel safer.

In 2004, the Advocacy program assisted 75 individuals.

Clients reduce emergency room use.

30

Task 1: Choose the Outcomes You Want to Measure

Hints:

Be thoughtful!! Define Your Target Population. Engage staff and clients in the discussion. Choose meaningful/compelling outcomes. Write outcomes so “John Doe” will be able to

understand. Develop a Logic Model.

31

Logic Model: A Picture of how a Program Works

Inputs Activities Outputs Outcomes

$

Staff

Outpatient Clinic

Referring agencies

Individuals with mental illness

Outside environment

Reduce level of incarceration of adults with mental illness

Reduce level of hospitalization of adults with mental illness

Psychiatric Services

Pharmacy Services

Nursing Services

Housing Assistance

24 hour crisis hotline

Vocational Services

75 psychiatric sessions

150 drug prescriptions

65 subsidized housing applications

175 crisis calls

80 resume writing/job placement sessions

32

Levels of Outcomes

In most cases, a program attempts to affect not one, but a series of changes for individuals, each building on the previous one. These can be thought of as the stepping stones toward the ultimate, or long term goal of the program. We can define these as:

Initial Outcomes

Intermediate Outcomes

Long term Outcomes

33

Relationships among levels of outcomes

Levels of outcomes can be thought of in terms of a series of interrelated milestones. The achievement of one sets the condition which increases the chances of achievement of the next.

In other words, IF “a” happens, THEN “b” is more likely to follow.

34

Relationships among levels of outcomes

Example:

IF job seekers learn appropriate job interview skills (initial outcome), THEN they will use appropriate job interview skills in the interview (intermediate outcome). IF they use appropriate interview skills, THEN they will obtain jobs (long-term outcome).

35

Relationships among levels of outcomes

Similarly, it can work backwards: if you know the desired long term outcome for your program, you can ask yourself, “what will make this more likely to occur?”

Example: Intended benefit (Long Term Outcome) for Employment Program: Clients get jobs. What will make this more likely to occur? Answer (one of many possible answers): Clients learn and demonstrate appropriate job seeking skills.

** There should be a theoretical/logical (research or practice supported) link supporting causality between the various outcomes.

36

Initial Outcomes

Initial Outcomes-The most immediate changes and most heavily influenced by the program, and are typically expressed in terms of new knowledge, skills, or attitudes learned through the program.

37

Initial Outcomes

Examples of Initial Outcomes:

Program: EmploymentInitial Outcome: Clients learn appropriate job seeking skills.

Program: Adult Day ServicesInitial Outcome: Clients increase awareness of community resources.

Program: Outcome Measurement Initiative

Initial Outcome: Staff demonstrate proficiency with Outcome Measurement concepts.

38

Intermediate Outcomes

Intermediate Outcomes-A link between the program’s initial outcomes and long term outcomes, typically expressed in terms of changes in behavior connected to the new knowledge, attitude, or skill (initial outcome) learned through the program.

39

Intermediate OutcomesExamples of Intermediate Outcomes:

Program: EmploymentIntermediate Outcome: Clients utilize appropriate job seeking skills.

Program: Adult Day ServicesIntermediate Outcome: Clients increase utilization of public and private services in their community.

Program: Outcome Measurement InitiativeIntermediate Outcome: Staff apply Outcome Measurement concepts to programs.

40

Long Term Outcomes

Long Term Outcomes-The ultimate objective of the program, typically expressed in terms of changes in condition or status. When defining Long Term Outcomes, it is important to select only outcomes which the program can be reasonably expected to influence.

41

Long Term Outcomes

Examples of Long Term Outcomes

Program: EmploymentLong Term Outcome: Clients are hired.

Program: Adult Day ServicesLong Term Outcome: Clients generalize acquired skills to other settings.

42

Important note, Long Term Outcomes

Just because outside influences may affect achievement of an outcome, that does not mean the outcome is inappropriate. We can adjust for this by recognizing that outcomes will not be achieved 100% of the time.

As an example, consider the effect of the economy (outside influence) on an employment program. The state of the economy cannot rule out the outcome “clients obtain jobs”; rather, the existence of an outside influence means that we should expect to see less than 100% of the outcome achieved.

43

Important note, Long Term Outcomes

Long Term Outcomes are of greatest interest to DHHS and will be the primary focus of our efforts to define outcomes for each program.

44

Example

Suppose we identify the long term outcome, “Clients get hired”. Well, we don’t necessarily care if the “means” to that end are the initial outcome “clients learn job seeking skills” or “clients are connected to warm job leads generated by job developers”. Either initial outcome can lead to the long term outcome. Depending on the particular agency’s expertise, service delivery method, or client mix, they may prefer one “means” to the other.

45

Example

Another example would be the long term outcome, “clients are appropriate weight.” One program may choose to work toward this end by targeting the initial outcome, “clients are able to identify healthy food” and the intermediate outcome, “clients eat healthy food”, while another program may identify, “clients learn the benefits of exercise” as the initial outcome and “clients exercise at least 2x per week” as the intermediate outcome.

46

Implications of Different Means to a Common End

Therefore, you can consider the Long Term Outcomes identified for your program to represent the direction we want you to go in. You will be responsible for deciding the best way to get there.

In some instances, external factors prescribe program outcomes or processes. An example would be the new Federal Birth to Three Outcomes which will be required for all BT3 programs. Additionally, the growth in popularity of evidence based program design is putting increasing pressure on programs to adopt proven approaches to service delivery. These will likely play an increasing role in programs.

47

Task 2: Specify Indicators ForYour Outcomes

Hints: Outcomes and Indicators are not the

same thing. Indicators-Measurable approximations

of the outcomes you are attempting to achieve. They determine the extent to which the outcome has been met.

Indicators look at the outcome and ask, “How do you know?”

48

Task 2: Specify Indicators ForYour Outcomes

Think about the outcome of the program. Ask yourself:

How do I know this Outcome when I see it? What does it look like when this Outcome happens? What are the characteristics of someone who has achieved this Outcome?

49

Task 2: Specify Indicators ForYour Outcomes

Example:

Program- Recreation.

Outcome- Clients socialize with peers and non-disabled individuals.

Indicator- Number of integrated community activities participated in during the previous month.

50

Examples of Program Indicators

Program Indicator Examples

Employment ST: # and % of clients who learn job seeking skills

Int: # and % of clients who demonstrate appropriate job seeking skills

LT: # and % of clients hired

Outcome

Measurement Initiative

ST: # and % of staff scoring 75% or better on Outcome Measurement proficiency test

Int:# and % of Programs which have Long Term Outcomes defined by 12/31/06

LT: # and % of performance based contracts

Adult Day Services ST:# of new self care skills learned

Int:# and % of days in a month client demonstrates new self care skills

LT: # and % of clients with improved health status resulting from improved self care

51

Indicators, continued

Notice how Indicators decrease the ambiguity of determining whether the outcome has been met.

52

Task 3: Prepare to Collect Data on Your Indicators

Hints:

Identify data sources for your indicators

i.e., Records, General Public, Program Participants, Parents

Design Data Collection Methods

i.e., Review of Program Records, Questionnaire, Interview

53

Things to Think About when developing a data collection method

Collect info that you want to know the answers to (Don’t just do it to satisfy funder requirements!!)

Don’t reinvent the wheel (Look for instruments that have already been developed!!)

54

Hints:

Make it Easy to Tabulate Your Findings. If possible, computerize!!!

Calculate the # and % of clients achieving the outcome.

Look at how gender, race/ethnicity, income, etc. influenced achievement of outcomes.

Task 4: Analyze Findings

55

Task 5: Use Your Findings

“If the outcome measurement system is not used for helping the organization improve its services, the time and effort will have been wasted.”

- Outcome Management in NP Orgs , Urban Institute, 2001.

56

Today’s intended training “outcomes”, revisited

Initial: Agencies understand DHHS definition and application of OM concepts. Agencies understand the direction DHHS is headed with Outcome Measurement and their role in that process.

Has this been accomplished?

57

The role for Agencies

o DHHS will identify long term outcomes for some programs by the release of the 2007 RFP.

o As mentioned previously, DHHS will define the long term outcomes for each program. Agencies will need to think about how they will work toward that outcome. In reviewing proposals DHHS will look at how logically connected your initial and intermediate outcomes are to the Long Term Outcome defined for the program, as well as how you propose to use indicators and data collection methods.

58

The role for Agencies

Agencies can make suggestions for outcomes, indicators, and data collection based on their practices: let DHHS learn from you by showing us what you are currently doing. You may submit this information at any time, but will be asked for it as part of your FY 2007 submission.

59

The role for Agencies

DHHS will use your proposed strategies for outcomes, indicators, and data collection methods as a jumping off point as we consider establishing these for each program area.

60

Resources, References

United Way of America Outcome Measurement Resource Network www.unitedway.org/outcomes

Measuring Program Outcomes: A Practical Approach, United Way of America, 1996. (Available through website for $5)

Kellogg Foundation Evaluation Handbook & Logic Model Guide www.wkkf.org

Urban Institute (Publications by Harry Hatry) www.urban.org

Fiscal Policy Studies Institute (Mark Friedman)www.resultsaccountability.com