Preparation For Outcome Monitoring 2010

transcript

Preparation for Outcome Monitoring

Presenters: 1. Jeff Wagers: Tx DSHS Program/

Training Specialist2. Ken Ripperger-Suhler, Ph D:University of Texas at Austin,

Researcher

Introductions

Presenters Audience Agenda Objectives

What is an Outcome?

The resulting changes in the client during or after a client attends an intervention.

Brainstorm-Outcomes!

1. What outcomes would you like to see in your

intervention?2. Each intervention will report

Agenda Review Basics of Interventions and OM Walk through a sample intervention from

logic model to analysis Readiness to do Outcome Monitoring Small Group Breakout by Intervention Administering surveys in a trusting

atmosphere Practice inputting survey data Data analysis and Implications Final questions and wrap-up

Objectives

Review basics of interventions and OM Discuss parameters and preparation for

OM in Summer 2010 Examine and assess OM pre/post surveys

by intervention Learn how to input and analyze data Brainstorm challenges and solutions to OM Reassess readiness to do OM

Evidence-Based Interventions (EBI)

Clearly defined audience, goals & objectives Based in behavioral theory Focused on specific risk behavior Could be an individual, group, or

community level intervention

Evidence-Based Interventions (EBI)

Individual-Level Interventions: CRCS

Group Level Interventions

Community Level Interventions

Key Terms Internal Logic Core elements Fidelity Behavioral Determinants/ Factors that

Influence Behavior/ Risk Factors Tailoring Adapting Reinventing

Behavioral Determinants; Factors Influencing Behavior (FIBs)

The “why” Reasons a client or

population is engaged in the risk behavior

Come from the client through: CRCS: asking clients

questions GLI, CLI: community

assessment Interventions focus on

the BD/FIB as well as the behavior

Program Evaluation

• What is evaluation?

• Why evaluate?

Program Evaluation Systematic, consistent collection of data Shows program effectiveness Accountability Program Improvement Increases Capacity Defines parameters Identifies unmet needs Process vs. outcomes

What is Process?

“How” you provide your intervention

Fidelity to the intervention Quality Assurance issues Skill level of agency staff Numbers (who attended, how many

sessions, # of target population, etc)

Process Questions

Is the location accessible? Is the program implemented as designed? Was the curriculum followed? How many of the target population attended

the sessions? How well is the RRS able to keep the session

focused on risk behavior? Was staff culturally competent? Others?

What is an Outcome?

The resulting changes in the client during or after a client attends an intervention.

Outcome Questions Did the client

Learn anything? Improve their skills? Change their behavior? Realize they are at risk of getting HIV?

Outcomes measure:

S - skillsK - knowledgeA - attitudeB - belief

B - behavior

Logic Models

Problem Statement

Risk Behaviors

Influencing Factors

Intervention/Activities

OutcomesImmediate Intermediate

Outcomes

Immediate Outcomes: Immediateresults of the intervention, such as changes in knowledge, attitudes,beliefs, and skills.- Increased perception of HIV risk- Increased condom use skills- Increased self efficacy to use condoms

Intermediate Outcomes: Intervention Results that occur some time after the intervention is completed, such as changes in behaviors, policies, and environmental barriers.

-Increased condom use-Decreased number of partners

STD rates are high among

young Hispanic MSM

Problem Statement

Young Hispanic MSM don’t

use condoms

Young Hispanic MSM have multiplepartners

Risk Behaviors

Young Hispanic MSM lack skills to properly

use condoms

Young Hispanic MSM don’t know how to

negotiate condom use with

their partners

Influencing Factors

Factors Influencing Behavior

It’s the norm for young Hispanic MSM

to have a lot of sex partners

STD rates are high among

young Hispanic MSM

Problem Statement

use condoms

Risk Behaviors

Young Hispanic MSM lack

skills to properlyuse condoms

Young HispanicMSM don’t know how to

negotiatecondom use with

their partners

Safe-sex negotiationscenarios &role plays

Self-assertionskills training

InterventionActivities

Influencing Factors

Intervention Activities

Young Hispanic MSM have multiplepartners

Condom skills demonstration and practice

Immediate and Intermediate Outcomes

Young Hispanic MSM lack

skills to properlyuse condoms

know how tonegotiate

condom use withtheir partners

Influencing Factors

Safe-sex negotiationscenarios &role plays

Self-assertionskills training

InterventionActivities

Immediate Outcomes

Young Hispanic MSM know how to

negotiate condom use with

their partners

Young Hispanic

MSMuse

condoms

Intermediate Outcomes

Condom demo and practice

Young Hispanic MSM know how to use condoms

Problem Statement

Risk Behaviors

Influencing Factors

Intervention/Activities

OutcomesImmediate Intermediate

Impacts

Impacts: Long-term results ofone or more interventions overtime, such as changes in HIV infection, morbidity, and mortality

- Decreased HIV rates

Outcome Monitoring

Data CollectionAnd

Analysis

Steps to Outcome Monitoring Develop *SMART outcome objectives

Specific, Measurable, Appropriate, Realistic, Time-Based

Develop/ review/ modify surveys or other methods that collect outcomes

Gathering survey/ method data Input data into spreadsheets Analyze and report data (Outcome

Monitoring) Make program adjustments if needed

Example of Process ObjectiveProcess objective: “Clients will complete

SISTA”

Revised: “By December 31, 2010, 50 African-American women will complete SISTA as documented by client data system.”

Example of Outcome ObjectiveOutcome objective: “Clients who complete

SISTA will show an increase self efficacy”

Revised: “75% of participants who complete SISTA will increase self-efficacy to correctly put on a condom as measured by pre/post survey.”

Ways to Collect Outcomes

Collect Outcomes Skills

Correct condom use, needle-bleaching Negotiation, Communication, Assertiveness

Questions Knowledge Attitudes Beliefs Behaviors

Demonstrations and PracticeRole Plays

Measuring Skills

Demonstrations Examples:

Condom Use Skills Needle Bleaching Skills

Checklists Consistent Provide “clean” pre-test Demonstrate correct condom use Practice Next session, do a post-test and compare

results

Role Plays Examples:

Condom or bleaching negotiation skills Self Assertiveness skills

Observations: Consistent Provide “clean” pre-role play (or reverse role

play) Demonstrate correct method Practice Next session, do a post-role play and compare

results

Types of Questions Open Ended Closed Ended

True/False or Yes/No Fill in the Blank Likert Scale Frequency Indicator Multiple Choice

Question Answer Options

Answer Option Answer Code

True/False T/F

Yes/No Y/N

Fill in the Blank FB

Scale S

Frequency Indicators

Open Ended OE

Multiple Choice MC

Sample Questions Perceived susceptibility

“Given my current behavior, I am worried that I might get HIV” (T/F) (Y/N)

“Given my current behavior, how worried am I that I might get HIV? (Scale)

1 2 3 45

Not worried Very worried

Sample Questions Expected outcome related to condom

“I will get HIV from my main partner without using a condom” (T/F) (Y/N)

“How likely is it that I will get HIV from my main partner without using a condom?”

1 2 3 45

Not likely at all Very likely

Steps to Outcome Monitoring Gathering survey data

Individual interviews Written questionnaires Focus groups

Input data into spreadsheets Analyze and report data (Outcome

Monitoring) Make program adjustments

Putting It All Together

Example-VOICES/VOCES

VOICES/ VOCES is a one-session group level intervention for HRH men and women in which groups watch culturally appropriate videos of condom negotiation, condom use and assertive communication, and practice.

Participants role-play condom use scenarios and practice putting on condoms

Logic Model

Increased Condom

Increased Condom Use Knowledge

Increased Condom Use Self-Efficacy

Increased Condom Use

Intention

INTERMEDIATE OUTCOMES(BEHAVIOR)

IMMEDIATE OUTCOMES

ACTIVITIESINFLUENCING FACTORS

Practice Condom Use;

Distribute condoms

ViewVideos;Condom

Negotiation Role Plays

RISK BEHAVIORS

Unprotected Sex

Condom Use Knowledge

Condom Use Self-Efficacy

Condom Use Intention

At least 40% of clients will demonstrate greater knowledge of condom usage by identifying on a written test at the end of the session the correct answer (yes or no) with regard to the following knowledge questions:

a. Are specific lubricants appropriate for use with condoms;

b. Whether or not to unroll a condom before putting it on a man’s penis.

Objective (Knowledge of Condom Use)

Survey Items (Knowledge of

Condom Use)

Yes NoDon’t Know

1. Vaseline and lotion are good lubricants for condoms/rubbers.

2. You should unroll a condom before putting it on a man’s penis.

Spreadsheet for storing data

Client ID

EthnicAge at intake Gender

Sexual Orient

HIV Status

Pre ConKnow

A01 AA 30 M STR - yes yes

A02 AA 25 M BI - yes yes

A03 H 27 M GAY - yes yes

Frequencies Pre & PostItem Vaseline and lotion are good

lubricants for condoms/rubbers.

Yes(%)

Don’t Know(%)

(PRE) 20(100%)

(POST) 0(0%)

20(100%)

Vaseline and lotion are good lubricants for condoms/rubbers.

Post Responses

Yes No Don’tKnow

Pre Yes 0 20 0 20(100%)

Responses No 0 0 0 0(0%)

Don’t Know 0 0 0 0(0%)

Total 0 20 0 20(0%)

Frequencies Pre & PostItem You should unroll a condom before

putting it on a man’s penis.

Yes(%)

Don’t Know(%)

(PRE) 20(100%)

(POST) 20(100%)

You should unroll a condom before putting it on a man’s penis.

Post Responses

Yes No Don’tKnow

Pre Yes 20(100%)

20(100%)

Responses No 0(0%)

Don’t Know 0(0%)

Total 20(100%)

20(100%)

Table 1. Change in Knowledge Scores from Pre to PostItem %

correct post

% change

d to correct

% change

d to incorre

% change

to correct answers

1. Vaseline and lotion are good lubricants for condoms/rubbers.

100 100 0 100

2. You should unroll a condom before putting it on a man’s penis.

0 0 0 0

Outcome Monitoring: VOICES/VOCES Report

Condom Use KnowledgeThe results of cross tab analysis find that 100% of clients showed increased

knowledge of appropriate lubricants for condom use.

0% of clients showed increased knowledge regarding when to unroll a condom to put on a man’s penis.

Planning for Outcome

Monitoring

Planning for Outcome Monitoring

Readiness for Outcome Monitoring

Done on an individual agency basis Stable implementation Data collection instruments

developed by Ken with agency input Outcomes may change over time

DSHS Parameters for Outcome Monitoring

1. Skills-building, knowledge, behavioral determinants ( AKA FIBs)

2. BD/ FIBs identified in your intervention based on research

3. Pre/post survey (unless an agency wants to do observations)

4. Based on logic model of intervention

3-4 outcomes for GLI

Some Differences in Data Collection…. CRCS

Intermediate and Immediate Outcomes Use questions from behavioral assessment No standardized survey One initial assessment, up to two

reassessments Driven by prevention plan and goals Designated outcomes Pre/post done on same individual Percentages for outcomes 15-50% Not all outcomes will apply to all individuals

Some Differences in Data Collection…. Group Level

Immediate Outcomes Pre/post tests Standardized survey 3-4 outcomes individualized per intervention Pre/post done on same individual Percentages for outcomes 40-75% Can measure any SKABB

Some Differences in Data Collection…. Community Level: 2 components

Group Component-measured first 2-4 outcomes Will measure M-Groups or training of peer

volunteers Standardized survey Pre/post; percentages 50-80% Knowledge, skill-level, able to communicate

very important Will be collected first stage

Some Differences in Data Collection…. Community Level:

Community Component-measured second 2-4 Outcomes Done during community assessment Pre/Post tests; Venue-based Mainly measures knowledge, attitudes,

beliefs, community norms Percentages up to 15% Gathered yearly Will be collected in second stage

Outcome Monitoring Surveys

Ask 3-4 questions per outcome 10-20 questions per survey Can measure skills through survey Knowledge can be a component Include key outcomes of intervention, in

line with behavioral determinants Consistent data collection

Facilitated Group Activity 11. CLI: Discuss group and community level

components of intervention

2. GLI: Discuss and review OM surveys by intervention

3. CRCS: Brainstorm problems and possible solutions to gathering OM data

Guided Group Discussion

Ways to Collect Outcomes1. Giving Instructions

2. Oral Interviews3. Written Questions

4. Focus Groups

Activity 2: Data Input and Analysis Input sample data into a spreadsheet

Sample data is located in packet Input into your laptop using the sample

spreadsheet (partner with someone if needed)

Analyze the data Spreadsheet calculates the data What can you determine from analysis?

Next Steps

Per intervention, brainstorm challenges and solutions to

Outcome Monitoring

Final Questions? Please complete

your course evaluations

Thank you for your participation!

Contact information Jeff Wagers, TxDSHS

Jeffrey.wagers@dshs.state.tx.us 512-533-3022

Ken Ripperger-Suhler, University of Texas Ken.ripperger-suhler@dshs.state.tx.us 254-624-1028

Preparation For Outcome Monitoring 2010

Documents