Erik Augustson, PhD, National Cancer InstituteSusan Zbikowski, PhD, Alere Wellbeing

Evaluation

Learning Objectives

• What is evaluation?• What are the benefits of evaluation?• The importance knowing goals of the program (from

different stakeholders perspectives) and how these goals will inform the evaluation.

• The components of a process evaluation.• The components of an outcome evaluation.

Body of evidence- Research

• Consistent body of literature – Curry et al., 1995; Zhu et al., 1996– Glibert et al., 2006; Tzelepis et al., 2011

• Consistent support across multiple populations– Demographics– Comorbid medical and psychiatric

• Consistent support within international contexts– Largely within developed countries

Body of evidence- Research

• Quit rates 1.5-2.0 times higher than self-help alone• Quit rates 1.1-1.6 times higher than medication alone• Abstinence rates 10-20%

- Meta-analysis ~13%• Dose response with increasing # of calls delivered

• **Further research on general effectiveness is not high priority**

“What is Evaluation?

”Program evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future program development.

What is Evaluation?

• Practical and feasible• Conduct within confines of resources, time, and political

context• Serve a useful purpose• Conduct in an ethical manner to produce accurate findings• Use results to make decisions and improvements

Why Evaluate Quitlines?

• To monitor progress toward program’s goals• To demonstrate that the quitline is effective• To determine whether the components of the quitline are

producing desired effects• To permit comparisons among groups, particularly among

those with disproportionately high tobacco use and adverse health effects

• Justify the need for further funding and support• To learn how to improve programs• To ensure that only effective programs are maintained and

that resources are not wasted on ineffective programs

Evaluation Planning- Steps

Goals and Objectives of the Evaluation

• A goal is the overall mission or purpose of the program– Promoting quitting among youth and adults

• Objectives are the results to be achieved by the program– Good objectives should be specific and measurable (SMART)

• Specific- specific goal or action to take place • Measurable- Quantifies amount of change to be achieved• Achievable and Ambitious- realistic but challenging• Relevant- how it relates to the program goals• Time-bound- time by which the objective is to be achieved

Stakeholders

• Identify key stakeholders:

Stakeholders are people and organizations who are invested in the program and interested in the results

Primary intended users of the findings (funding agency, tax

payers, partners)

Those served or affected by the program (tobacco users, advocacy groups, community

members)

Those involved in operations

(program staff)

Stakeholders

• Know the goals of the program from different stakeholders perspectives and how these goals will inform the evaluation

• Considering the needs of stakeholders will increase the likelihood that the findings will be accepted and used

Typical Evaluations

Exercise- Review Data Tables and Chart

• What about this data is useful? • Does this data suggest a problem/concern?• Can you think of possible contributions and causes• How might you assess this problem/concern further?

Does this data suggest a problem or concern

Can you think of possible contributions and causes?

Possible Contributors and Causes

How might you assess this problem/concern further?

Decreased Satisfaction

Thank You