Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs If you...

transcript

Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs

If you have not already done so, please connect to the audio portion of this Webinar:

1. Dial 1-877-668-4490

2. Enter the access code 622 071 435 #

3. Enter your unique participant ID number displayed

on your screen

OR choose the option to let the program call you back!!!

Housekeeping

Phones are muted Ask questions by

sending a chat message to “All Participants” using the chat panel on the right side of your screen

Who’s on the Webinar

Representing Programs and Organizations from 44* States

175 Attendees

Beginning with the End in Mind: What We Know about Performance Measurement for VMF

Programs

Outcomes

You will:a. Learn about Performance Measurement for Veteran

and Military Family Programs (VMFP)b. Discover methods to measure outputs and

outcomes with available toolsc. Learn about VMFP data sources to help measure

outputs and outcomes d. Explore best practice examples in measuring

outcomes from Equal Justice Works

Webinar hosts: Education Northwest

Hello and Welcome from CNCS!

Koby LangleyCorporation for National

and Community

Service

*Service to VMF*Creating Service/ Volunteer Opportunities for VMF

Education and Certification

Employment

Access to Benefits

Youth Mentoring

Transportation

Disaster Preparedness

Wellness and Other Support Services

*Community Coordination

* Overarching activities of programs in SAA

Specific activities Mentioned in SAA

The Serve America Act’s Influence on CNCS VMF Performance Measures

Session Presenters

Dr. Chris Spera, CNCS Directorof Research and Evaluation

Kerry O’Brien, Director of Federal Programs and Strategic

InitiativesEqual Justice Works

Kathryn Gravely, Program Manager, Federal Programs and Strategic Initiatives, Equal Justice Works

Key National Service SAA Issues Areas by Program

Wellness/Other Supports and Services by Program (n=59)

Performance Measurement for Veteran and Military Family Programs

• What is performance measurement?• A systematic process of tracking outputs and

outcomes based on the goals of your program, as articulated in your program’s logic model

• What is a logic model?• A logical chain of connections showing what the

program is to accomplish

Why is Performance Measurement important for CNCS?

• Assess the individual and collective results of programs and continue to enhance program effectiveness

• Expand CNCS’s ability to account for the combined contributions of all organizations making use of our resources

Why is Performance Measurement important for VMF Programs?

• Assesses the individual and collective results of VMF programs by tracking outputs and outcomes related to improving

• Military family strength and resilience

• Quality of life for veterans, i.e.,:

• Transition to civilian life

• Education attainment

• Job/vocational skills

• Behavioral health

Performance Measurement for VMFP: Collecting Data on Outputs and Outcomes

• Tools to measure outputs:• Intake forms, attendance and tracking sheets to keep count

• Tools to measure outcomes:• Pre, mid-term and post program surveys designed to

measure increases in specific areas over time, as a result of participation in the program, focus groups, interviews

• Where to find instruments, tools and resources for performance measurement:

• http://www.nationalservice.gov/resources/npm/home

• search by focus area

Performance Measurement for Veteran and Military Family Programs

• Performance measures track outputs and outcomes based on the goals of your program’s logic model

• A logic model shows the chain of connections illustrating plans to meet community needs; the theory of change

INPUTS OUTPUTS OUTCOMES

Program investments

Activities Participation Short Medium Long-term

What we invest:

People, time

Service provided:

# of in-take & referrals

Service recipients:

# of returning veterans receiving referrals

What results/changes:

Veterans placed in employment prep programs;

then employed

Need: access to employment resources

Outputs for Veteran and Military Family Programs

• Defining outputs • Amount of service provided (e.g., people served, products

created, or programs developed) through the service intervention of your program

• Common outputs for VMF programs• # of veterans or active Service members and families

served

• # of veterans or active Service members engaged in service provision through CNCS programs

• Others?

Outputs for Veteran and Military Family Programs – Program Examples

• Veterans Helping Veterans Now• # of veterans referred to service providers that can meet

their needs

• Metro Community College, Omaha, Nebraska• # of people who received outreach from VISTAs regarding

support needed by veterans returning to college

• CA Conservation Corps• # of veterans employed as crew members

Outcomes for Veteran and Military Family Programs

• Defining outcomes• Reflect the changes in attitudes/beliefs, knowledge/skills,

behavior, or conditions of individuals that occur in the organizations, communities, or environment as the result of the services provided

• Common outcomes for VMF programs• Increase in educational attainment

• Increase in vocational skills and credentials (e.g., professional licensure)

• Increase in employment outcomes

• Others?

Outcomes for Veteran and Military Family Programs– Program Examples

• Veterans Helping Veterans Now• # of veterans receiving services as result of referral who

then gain employment (for example) due to services received

• Metro Community College, Omaha, Nebraska• # of veterans succeeding in college due to support of

informed college staff and faculty

• CA Conservation Corps• # of veterans who completed the program and gained key skills

for future employment

Sources of Data Collection

• Self-reported data through national service members and volunteers (e.g., # of veterans served)

• Self-reported data from veterans through surveys (e.g., percentage of veterans served that obtain employment post-service)

• Public use and administrative data sources (e.g., data from local VA organizations)

• Observation – outside observers making ratings, counting behaviors

Self-Report Data from Service Members

• Key Question: Who are you serving?

• Self-reported data through national service members and volunteers

• Example: # of veterans served

• Example: # of military families served

• Important to know penetration rate (i.e., percent of veterans served in relation to total size of veteran population in your community)

• VA data source, called VETPop, has publicly available data source on veteran population by zip code

Self-Report Data from Beneficiaries

• Key Question: What outcomes are you achieving?

• Potential Outcomes:

• Increase in educational attainment

• Increase in vocational skills and credentials (e.g., professional licensure)

• Increase in employment outcomes

• Data Sources:

• Self-report survey data from veterans/military

• Public use data sources

Public Use Data Sources of Interest

• Demographic Data on Population• Veterans Population (VETPoP)

• DMDC (DOD) – demographics report

• Public Use Survey Data• Health (Wellness)• Survey of Army Families (SAF)

• Air Force Community Assessment Survey

• National Survey of Veterans

Public Use Data Source of Interest

• Public Use Survey Data (Cont.)• Health (Behavioral Health)

• Survey of DoD Health Related Behaviors

• MHAT Screenings

• Sexual assault survey

• Employment• VBA Vocational Rehab Survey

• Homelessness• VA/DoL homeless counts

Administrative Data Sources of Interest

• Administrative Data• Homelessness

• DoL VETS data on HVRP data

• Education• Data on GI bill usage (VA)

• Vocational Rehabilitation data

• Employment• Transition Assistance Program data

• Local data sources from DVOPs, LVERs, etc.

Tools and Resources for Performance Measurement

http://www.nationalservice.gov/resources/npm/home

Equal Justice Works

Kathryn GravelyProgram Manager

Kerry O’BrienDirector of Federal

Programs

Equal Justice Works Overview

• Mission– Equal Justice Works is a 501(c)(3) nonprofit organization

dedicated to creating a just society by mobilizing the next generation of lawyers committed to equal justice

• AmeriCorps Program– 40 full-time lawyer members in 10 states. Most members

serve for two years

– 360 minimum-time law student members in 48 states. Most members serve 300 hours in the summer

– Placement sites are legal aid organizations and courts working with low-income residents, including veterans

Equal Justice Works: Veterans Theory of Change

Summary

Need: Veterans have complex legal problems that keep them homeless and in poverty

Intervention: Lawyers and law students, in conjunction with veterans service infrastructure, resolve legal problems

Evidence: 2012 Evaluation, Studies show that those with lawyers fare better than those without

Outcome: Lives of veterans are improved, increased income, access to health care, better housing

• Need– Low-income and homeless veterans face complex legal

problems related to disability benefits, fines, warrants, housing, employment, debt, and family issues

– Lawyers are uniquely equipped to resolve such problems

• Data Documenting Need– 2011 V.A. CHALENG report; Research articles on

veterans & need for lawyers; HUD, HHA and Veterans Benefits Administration statistics; PTSD and TBI studies (e.g. 3 of top ten unmet needs of homeless vets are legal needs – CHALENG)

• Intervention– Lawyers and law students will resolve the legal problems

of low-income and homeless veterans, leading to improved lives: higher incomes, better health, safe and secure housing and family stability

• Intended Outcome– Improve the lives of veterans

• Evidence-basis for Intervention– Impact of 2010-2012 Veterans Pilot Project

– Studies that lawyers make a difference

Equal Justice Works: Outputs vs. Outcomes

• Overarching Goal– Through legal assistance, improve the lives of

veterans

• Outputs: Amount of Service Provided

– Example: Assist 30 veterans with benefits claims

• Outcomes: Reflect the Changes that Occur

– Example: 20 veterans win benefit cases, providing access to health care and $360,000 in additional yearly income

Equal Justice Works: Outcomes

Income Employment Housing Other

VA Disability Benefit cases won

$ benefits received

% increase in income

Fines/Penalties waived

# of Minor Criminal Records Sealed or Expunged

# Employed after legal barrier removed

% increase in wages

# Vets housed after legal barrier removed

# Vets remaining in housing

Access/better access to health care

Discharge upgrades (leads to income, housing, access to health care)

Wills for the dying

Equal Justice Works: Data Collection

• What we could say in 2010– 1,500 veterans served

– Compelling stories about individual cases

• What we could not say in 2010– Outcomes of cases

– Impact on veterans served

– Specific information on veterans served (e.g., demographics, legal issues addressed, etc.)

– Which sites were performing well/needed assistance to meet goals

• Today, we can say: – 1,500 veterans served

– Members completed 226 thorough case evaluations

– Filed 91 disability benefits petitions or appeals

– Thus far have won 29 of these cases, securing $642,668 in back benefits and $430,933 in yearly benefits for their veteran clients

– 114 benefits cases are still pending– Successfully able to remove a barrier to employment for 127 veterans served

– Demographic information specific to veterans served (e.g., era of service, age, gender)

– Number of veterans served for each particular legal issues, both on macro and micro levels

• Today, we can say: – Fines or other penalties waived or reduced for 78 veterans

served, totaling $36,771

– Assisted 19 veterans in getting their criminal matters dismissed or expunged

– Assisted 118 veterans in repairing their credit

– Of the 1,416 veterans served, 740 were homeless or at risk of becoming homeless, and 739 were veterans with a disability

– 77% were male and 23% were female—meaning that Fellows served a high number of female veterans, as 10% of veterans in the U.S. are female

• Today, we can say: – 20% of those receiving services served in the Vietnam era,

and 23% served in Iraq and Afghanistan

– The remaining veteran clients served in all other eras of service, including pre-Vietnam, the Gulf Wars and during peacetime

– The majority of veterans served were either between 26 and 35 years of age or 55 and older (216 and 420 served, respectively)

• 2011– Conducted interviews and focus groups about outcomes

• 2012– Implemented new optional measures early 2012

– Brought supervisors together in person – October

– Outside evaluation completed – December

• 2013 – Data resulted in stronger application for funding – Feb

– Implemented refined measures & made them mandatory in March 2013

– Report submitted May 1 substantially better

• Collection Format: – Was: Custom-built web application

• Cost an hourly charge every time data needs changed• Was getting expensive• Output was Excel spreadsheet

– Now: Google Drive - Forms• Program manager and coordinator designed, refined• Output is Excel spreadsheet• Drive.google.com – Create “Forms”

Equal Justice Works: Data Collection in Google Forms

Equal Justice Works: Great Stories

– Equal Justice Works AmeriCorps Legal Fellow Rochelle Richardson

– Homeless Persons Representation Project in Baltimore, MD

– Many years experience as an employee of the VA

– Reexamine the case of Mr. A, a homeless veteran with a service-connected mental disability whose claim for veteran's disability benefits had already been denied

– Rochelle obtained records that had not previously been reviewed and petitioned to reopen the claim and submit further evidence

– The petition was successful & the claim was granted in March 2013

– The client will receive a retroactive one-time payment of $50,000 and $3,000 a month in benefits

Equal Justice Works: Lessons Learned

• Define intended outcomes early

• Align reporting with what funders require

• Have performance measures and data collection tools ready at the beginning of program year

• Have a “steering committee” of on the ground experts to consult

• Keep refining and improving measures to more accurately reflect program impact

Equal Justice Works: Next Steps

• Figure out how to track and capture long-term outcomes – Example: Did increased income result in better housing,

better health, better outcomes for children of veterans served?

• Comparison Evaluation– Compare veteran outcomes in our program with national

statistics or other programs

• Big Vision– Randomized control trial

Performance Measurement for Veteran and Military Family Programs: Re-Cap

• Performance Measurement:– Tracks outputs and outcomes based on the goals of your

program to meet community needs

• Outputs: – Amount of service provided (e.g., veterans) through

the service intervention of your program

• Outcomes: – Reflect the changes that occur as the result of the

services provided (e.g., veterans employed)

Questions???

Dr. Chris SperaCNCS

Kerry O’BrienEqual Justice Works

Kathryn GravelyEqual Justice Works

To Continue this Discussion…and Others

Veterans and Military Families

Knowledge Network

www.nationalserviceresources.org/veterans

http://bit.ly/VetsNet

Join Us for the Next Webinar in this Series

June 12, 2013

Making a Difference: What We Know about Using an Evidence-based Approach

Register: http://tinyurl.com/c8gp8ll

Thank you!Please complete the

post-webinar survey

Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs If you...

Documents