Developing an Evaluation Plan for a Comprehensive Tobacco Control

Program

Jesse Nodora, DrPHMichele Walsh, PhD

Joanne Basta, PhD

Learning Objectives

• Describe what an evaluability assessment is and the steps involved for conducting one in order to inform an evaluation plan

• Describe the key components of an evaluation plan for a comprehensive tobacco control program

• Recognize the major challenges and rewards of evaluation planning

Presentation Outline

• Overview of TEPP, TEPP Evaluation History, and Arizona Population (background & context)

• TEPP Evaluability Assessment

• Key Findings

• TEPP Evaluation Plan Components

Arizona Demographics

• Population 5,456,453• Race/Ethnicity

• Non-Hispanic White 64%• Hispanic 25%• American Indian 5%• African American 3%• Asian & P. Islander 2%

• U.S. Census 2002 Estimates

Arizona Adult Smoking Prevalence Estimates: 1990-2002

0102030405060708090

100

Per

cen

t

AZ BRFSSATSNational

AZ BRFSS 20.6 23.8 19.2 20.9 23 22.9 23.7 21.1 21.8 20 18.5 21.5 23.4

ATS - - - - - - 23.8 - - 18.8 - - 20.1

National 25.5 25.7 26.5 25 25.5 24.7 24.7 24.1 23.5 23.3 22.8 23

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

Arizona High School StudentsSmoking Prevalence Estimates: 1991-2003

0102030405060708090

100

1991 1993 1995 1997 1999 2000 2001 2002 2003

AZ Substance Abuse Survey Arizona Youth Tobacco Survey

National Youth Risk Behavior Survey Arizona Youth Survey

Legacy Foundation National YTS

TEPP Values

TEPP is committed to excellence through integrity and inclusiveness, honoring individual, cultural and religious practices.

We value the communities we serve:

Through excellence in developing quality programs and services that are determined by community needs and are data driven.

By acting with integrity, being truthful and showing compassion.By seeking and integrating community input through creating

partnerships, collaborations and accessible services.By giving respect and dignity with a commitment to honoring

individual, cultural and religious practices.

TEPP Mission

Based upon a belief in excellence, inclusiveness and integrity that honors individual culture and religious practices, the TEPP mission is to protect and improve the health and quality of life of all Arizonans by reducing tobacco use through prevention and treatment, by denormalizing tobacco use and by reducing exposure to environmental tobacco smoke.

TEPP Goals

• TO REDUCE MORBIDITY AND MORTALITY RELATED TO TOBACCO USE

• TO PREVENT AND REDUCE TOBACCO USE AMONG ALL ARIZONANS

• TO PROVIDE AND SUPPORT ACCESSIBLE, AFFORDABLE AND EFFECTIVE CESSATION SERVICES AND SYSTEMS

• TO REDUCE ALL ARIZONANS' EXPOSURE TO SECOND HAND SMOKE (SHS)

TEPP Goals-Cont.

• TO ENSURE INCLUSITIVITY OF POPULATIONS WITH TOBACCO CONTROL DISPARITIES IN PLANNING AND SERVICES

• IMPLEMENT AN INTEGRATED, COMPREHENSIVE EVALUATION MODEL THAT PLANS, SURVEYS AND EVALUATES PROGRAM PROCESSES AND OUTCOMES

TEPP Program Components

• Community Programs• Marketing and Public Relations• State-wide Programs• Evaluation• Administration

TEPP Expenditures

• FY 1995/96 $9.7M (76% M&PR 17% CP)• FY 1996/97 $18.2M (64% M&PR 17% CP)• FY 1997/98 $24.2M (55% M&PR 26% CP)• FY 1998/99 $32.5M (43% M&PR 33% CP)• FY 1999/00 $34.5M (41% M&PR 38% CP)• FY 2000/01 $36.6M (33% M&PR 40% CP)• FY 2001/02 $31.3M (28% M&PR 44% CP)• FY 2002/03 $28.6M (23% M&PR 35% CP)

M&PR - Marketing and Public RelationsCP – Community Programs

TEPP Evaluation History

• 1996-1998 • Adult and Youth Tobacco Surveys • Open Systems Process Evaluation• Media Tracking and Surveys• LP Assessments (adult cessation, prevention)

• 1999-01• Adult and Youth Tobacco Surveys • Open Systems Process Evaluation End• Policy Surveys (Schools, Worksites, Ordinance

Tracking, Policymaker Perceptions)• 2002-03

• Adult and Youth Tobacco Surveys • TEPP Evaluation Unit

TEPP Evaluation Unit & ADHS

• Semi-independent Unit• Addressing comprehensive

evaluation (Links among data)• Addressing all TEPP programs• Evaluation as a developmental

process

Program planning

Program implementation

Evaluation

Learning

Perspective on Program Evaluation

Evaluation is a developmental process rather than a single event or periodic report card

Evaluability Assessment - Defined

“An evaluability assessment is a process for clarifying program designs and exploring a program’s context in order to determine if it can be meaningfully evaluated.” (Wholey, Hatry & Newcomer, 1994)

Recognizes that programs, even at same site, are at differentlevels of development

Five-Tiered Approach

II

IIII

IIIIII

IVIV

VV

Needs Assessment

Monitoring and Accountability

Quality Review & Program Clarification

Achieving Outcomes

Establishing Impact

Organizes evaluation around these developmental levels

Adapted from Jacobs et al., 2000

Allows Mapping to Evaluation Types

• Needs/Resource Evaluation• Knowing what programming is needed

• Process Evaluation• Knowing what programming is happening

• Outcome Evaluation• Linking program to short-term changes

• Impact Evaluation• Linking program to longer-term changes

• Cost Analysis Evaluation• Are the results we are producing likely to be

cost efficient?

Evaluability Assessment

Answers these basic questions:

Answers these basic questions:

What evaluationactivities best support program enhancement?

What evaluationactivities best support program enhancement?

What level and type of evaluation best suits program and stakeholder

needs?

What level and type of evaluation best suits program and stakeholder

needs?

Evaluability Assessment - Steps

• Examined program documents• Reviewed past evaluations• Distributed pre-site visit

questionnaire (community pgms)• Conducted site visits• Statewide project interviews• Identified existing data sources

General Findings

• Considerable variability in programming across the state

• Considerable amount of existing data

# Data Source Name

Overview Description Owner Data Collection Dates

Who Collects Data

How Respondents are Selected

Method of Data Collection

Where Data Reside

Surveillance 1 Adult Tobacco

Survey--ATS Was initiated in 1996 by the Arizona Department of Health Services (ADHS) to collect detailed surveillance of adult tobacco use and attitudes. The survey has since been repeated in 1999 and 2002 to provide evaluation data for the Tobacco Education and Prevention Program (TEPP) through continual monitoring of tobacco use among Arizona adults.

ADHS 1996 (baseline) 1999, 2002. Every three years.

Social Research Laboratory at Northern Arizona University; ADHS Telephone Survey Center (1996, 1999)

Disproportionate stratified sampling, random digit dialing that over-represents rural areas. Respondents must be: AZ resident, 18 or older; speak English, and have a phone. 1996 and 1999 surveys had English and Spanish versions. There will be a Spanish language survey launched in 2003.

Computer Assisted Telephone Interviewing (CATI)

ADHS-TEPP Evaluation Unit

2 Behavioral Risk Factor Survey--BRFS

A joint effort between CDC and ADHS to collect annual data on health risk behaviors of non-institutionalized residents 18 years of age & older. Used to plan, implement, and monitor health promotion and disease prevention efforts in AZ. Core questions include items about tobacco use.

ADHS & CDC

Annually since 1982

Private firm contracted by ADHS

Disproportionate stratified sampling using random digit dialing. Residents, 18 years and older, non-institutionalized, have phones.

Computer Assisted Telephone Interviewing (CATI)

ADHS

3 Youth Tobacco Survey--YTS

Developed by the CDC for state surveillance and initiated by ADHS in 1997 to collect detailed surveillance of youth tobacco use and attitudes. The survey has since been repeated in 2000 and 2003 to provide evaluation data for the Tobacco Education and Prevention Program (TEPP) through continual monitoring of tobacco use among Arizona adults. Developed by the CDC for state surveillance.

Arizona Department of Education; ADHS/TEPP

1997 (baseline), 2000, 2003. Every 3 years.

For 2003-ORD-Macro contracted by Arizona Department of Education

For 2003-The YTS survey sample design is developed by the CDC and is a 2-stage cluster design. In the first stage, schools are selected randomly within the grade range specified with a probability proportional to enrollment size. At the second stage, classes are randomly selected from within the selected schools and all the students within a selected class are surveyed. Included grades 6-12 in both public and charter schools. Data are weighted to provide state prevalence estimates.

2003 survey administered to youth in schools. 1997 was a phone survey of youth ages 10-17. 2000 survey was administered in schools to grades 6-8, and 9-12 in public and charter schools. Grades 9-12 data were not released due to low response rate.

ADHS-TEPP Evaluation Unit has 2000, and 2003 data.

4 Tribal Youth Tobacco Survey

Developed by the CDC for state surveillance specifically focusing on Native American youth and initiated by ADHS and Inter Tribal Council of Arizona to collect detailed surveillance of Native American youth tobacco use and attitudes.

Individual tribal nations in Arizona and Native American Urban Health Centers

Varies Individual Arizona tribal nations or Native American organizations with assistance from CDC.

Survey administered in schools.

Individual Arizona tribal nations and Inter-Tribal Council of Arizona

5 Youth Risk Behavior Survey--YRBS

Developed in 1990 by CDC to monitor priority health risk behaviors that contribute to leading causes of death, disability, and social problems among youth and adults in U.S. Includes items about tobacco use. Provides comparable national, state and local data, and among subpopulations among youth. Monitor progress toward achieving Healthy People 2010 objectives. Voluntary state participation.

Arizona Department of Education (ADE)

2003—first time Arizona participated in YRBS. Conducted every three years.

ORD-Macro contracted by Arizona Department of Education

9th-12th grade in Arizona public and charter schools are administered the survey using CDC sampling design protocol.

Survey administered in public and charter schools to 9th-12th grade students.

Arizona Department of Education

6 Arizona Youth Survey--AYS (formerly the Arizona Youth Substance Abuse Survey)

Survey of 8th, 10th, and 12th grade students in Arizona, to fulfill the legislative requirements to conduct a survey to measure both the attitudes and actual prevalence and frequency of substance abuse by children and adults. The survey is a partnership between the Arizona Criminal Justice Commission (ACJC) Arizona Department of Health Services, and Governor’s Division of

Arizona Criminal Justice Commission (ACJC)

Biennial administration. Last survey conducted in 2002.

Southwest Center of Prevention at University of Oklahoma-contracted by ACJC

Stratified random sample based on school size, and a weighting procedure to accurately represent student composition in 8th, 10th, and 12th grades. Schools can voluntarily participate if not selected for sample, but data are not reported in official

Survey administered in public and private schools.

Arizona Criminal Justice Commission (ACJC)

Youth Access Counter Acts Pilot Study Data

Study that assesses the impact of the CounterActs Program on tobacco merchants. Maricopa County intervention site and Pima County control site. Baseline and post survey using cross-section sample to assess public’s perceptions about youth access. Random compliance checks conducted by law enforcement agents and post-compliance check questionnaire administered to store clerks.

ADHS/TEPP 2002 to 2003 Contractor for ADHS supervised by Eusebio Alvaro, U of A. Attorney General Special Agents

For cross-sectional survey: Random sample of adult Maricopa County residents to assess perceptions about youth access to tobacco. For compliance checks: merchants and/or clerks who were inspected.

Pre-post cross-sectional survey Computer Assisted Telephone Interviewing (CATI) For compliance checks: merchants and/or clerks who were inspected given short questionnaire to complete.

Attorney General Compliance Inspection Reports

A database that holds descriptive data about each individual inspection and its outcome that was conducted by the Attorney General regarding merchant compliance with the youth tobacco access law.

ADHS/TEPP Ongoing since 2002

Attorney General, Special Investigations Section Agents

Decision-making and criteria determined by Attorney General’s office and ADHS/TEPP.

Special Agent completes the inspection form after each merchant inspection.

Attorney General’s office; ADHS/TEPP Evaluation Unit

Synar Database A database that holds information about the outcome of the Synar inspections. By law, states must conduct random unannounced inspections to determine buy rate of tobacco products sold to youth under the age of 18. State rate must not exceed 20% otherwise state may lose 40% of its substance abuse block grant.

ADHS Annual since 2000

Contractor for the State to conduct inspections

Random, unannounced inspections of retailers who sell tobacco.

Attorney General’s Office; ADHS TEPP Evaluation Unit

School Based Prevention & Other Youth Data Arizona Department of Education –School Report Cards

Information about Arizona Schools such as resources that each school has; indicators of risk at the school such as free/reduced lunch, attendance rates, test scores, expenditures; safety and healthy environment

ADE ADE

Arizona Department of Education—Safe and Drug Free Schools Survey and Report (formerly CHAPPS)

Part of the Comprehensive Health Education Standards Surveillance System (C.H.E.S.S.S.) Purpose is to account for prevention dollars at the site/school level, and is part of a collection system to fulfill data reporting requirements to state and federal agencies. Includes data about tobacco prevention programming and policy enforcement.

ADE Annual since 1998

ADE, Research and Policy Division

Administrative personnel at all public and public-charter schools must complete the survey.

Web-based survey hosted on ADE website.

ADE

Arizona Department of Education—School Health Profiles (AZ SHEP)

Assesses the status of school health programs and policies among samples of middle/junior high and senior high schools in CDC funded states and selected cities. For example, what health topics are covered, teaching methods used, school policies on health-related topics, such as tobacco.

ADE Annual since 1996?

ADE School principal and lead health educator at the sampled middle/junior high and senior high schools in Arizona.

Self-administered questionnaires. Voluntary and confidential.

ADE

TEPP Local Project School Based Prevention Pre/Post Surveys

To assess the outcomes of the TEPP Local Project school based prevention programs. Different surveys developed by various local projects to assess changes in knowledge, attitudes, intentions, and behaviors of program participants who received intensive school-based prevention.

Various TEPP Local Projects

TEPP Local Project Staff and/or School Teachers.

Participants in TEPP school-based prevention programs.

Administered to students in schools.

Individual Local Projects. Some data housed in TEPP Evaluation Unit (e.g. Pima County, Coconino County

Arizona School Tobacco Policy Survey

Two surveys conducted of Arizona public schools to document whether schools had written policies or rules regarding tobacco use. The 1998 survey was done prior to passage of ARS 36-798.03 prohibiting tobacco products on school grounds, in buildings, etc. The 2000 survey was designed to assess compliance with the new law.

ADHS/TEPP 1998 and 2000 Arizona Cancer Center, The University of Arizona

All public, alternative, and charter schools in Arizona

Computer Assisted Telephone Interviewing (CATI). Two stage calling: 1) to set appointment with appropriate person; 2) to conduct interview

Arizona Inter- Data from outcome and satisfaction survey ADHS/TEPP Ongoing AIA staff All participants in the AIA Pre-post program ADHS/TEPP

General Findings

• Need for a phased approach to develop meaningful evaluation plan

• Steps required• Establish evaluation infrastructure• Create structured process evaluation to clarify

and quantify program• Identify appropriate outcomes at different

levels• Design reporting that supports program

planning

Evaluation Infrastructure Components

• Develop coordination model

Synthesis of scientificliterature

Reports and presentations

Education regarding evaluation methods

and procedures

Measurement development

Data management

Workshops

Refining program goals and objectives

Training materials

Database development

Measurement tools

Adherence to improved data quality control procedures

Increased knowledge about and skill in evaluation

methods and procedures

Increased recognition of evaluation

utility

High data quality

Appropriate data interpretation

Integration of evaluation and program planning

and development

Decreased tobacco-related disparity

Reduced tobacco-related morbidity and mortality

Reduced smoking prevalence

Effective comprehensive tobacco

control programs

Input Activities Short-term IntermediateOutputs Long-term

Outcomes

Coordinating Evaluation Efforts with Local Projects

Local project

staff

Evaluation UnitStaff

LPLogic

models

Needs and

resource assessment

LPplans

ADHS TEPP

strategic plan

ADHS TEPP staff

Data analysis

Positive partnership between

evaluation unit and local projects

EvaluationStandards

Implementationof

standardizedmeasurement procedures

Consistent communication with

local project staff

Increased understanding of roles and responsibilities

in evaluation of comprehensivetobacco control

Increased knowledge of science-based practice for

tobacco control

Links with statewideprojects

Evaluation Infrastructure Components

• Develop coordination model• Assess feasibility of centralized IT for

evaluation

Process Evaluation Components

• Develop measures of program quantity

• Develop measures of program quality

• Coordinate with other state and national efforts• Promote comparisons across state

• Develop system with an eye towards linking to outcome

Outcome Evaluation Components

• Identify appropriate outcomes at different levels • Work from state and local program models • Use existing data where possible to build historical

picture• Examine proximal, rather than distal, outcomes

• Use appropriate measures• Standardize instruments used • Align with national and other state instruments

where possible • Go beyond surveillance data alone as outcome

• Use appropriate data analytic techniques• Incorporate multivariate models

Key Challenges

• Recognizing and accepting appropriate level of evaluation

• Process evaluation not as “sexy” as outcome• But need to know the “program” in program evaluation

• Identifying appropriate outcomes• Developing complementary measures to supplement

surveillance data• Coordinating needs of local project evaluations with

broader, statewide evaluation • local: short term outcomes, less rigorous, quick

turnaround• statewide: standardization of data elements and

collection procedures, rigorous design, intermediate to longer term outcomes, and a relatively long time-frame

• Creating (and retrofitting) evaluation systems• Longer process than anyone wants it to be

Key Rewards

• Unpacking the “comprehensive program” black box

• Clarifying program planning• Supports clearer thinking about goals and how

attempting to reach them• Encouraging better understanding of how

programs are linked to outcome• Anchoring a baseline for evolution of

program and evaluation planning• Facilitating intra-state and interstate

communication

ReferencesU.S. Census Bureau 2002, State and County Quick Facts. (http://quickfacts.census.gov/qfd/states/04000.html;

http://www.census.gov/statab/ranks/rank01.html)

Campaign for Tobacco-Free Kids, "A Broken Promise to Our Children: The 1998 State Tobacco Settlement Five Years Later,” November 2003. (http://www.tobaccofreekids.org/reports/settlements/2004/fullreport.pdf)

Jacobs, F., Kaouscik, J. Williams, P., & Kates, E. (2000). Making it Count: Evaluating Family Preservation Services. Tufts University Press.

Wholey J, Hatry H & Newcomer K. (1994) Handbook of Practical Program Evaluation. San Francisco, :Josey-Bass Inc.

For further information contact: Jesse Nodora, Dr.P.H. Or Michele Walsh, PhDEvaluation Administrator Senior Evaluator

Arizona Department of Health Services Tobacco Education and PreventionTobacco Education and Prevention Program Program Evaluation Unit1740 West Adams The University of ArizonaPhoenix, Arizona 85007 PO Box 210462 Tucson, Arizona 85721(602) 364-0837 Fax (602) 364-0844 (520) 318-7259 x 123jnodora@hs.state.az.us www.tepp.org mwalsh@u.arizona.edu