Research and Evaluation Framework Implementation Guide · 2017-04-27 · 2 Introduction Research...

Research and Evaluation Framework Implementation Guide

(2nd edition)

A guide to inform planning and reporting for health promotion programs

Visit us on the web for more information

http://ww2.health.wa.gov.au/Articles/A_E/Evaluation

© Department of Health, State of Western Australia (2017)

Copyright to this material is vested in the State of Western Australia unless otherwise indicated. Apart from any fair dealing for the purposes of private study, research, criticism or review, as permitted under the provisions of the Copyright Act 1968, no part may be reproduced or re-used for any purposes whatsoever without written permission of the State of Western Australia. Suggested citation:

Chronic Disease Prevention Directorate. Research and Evaluation Framework and Implementation Guide (2nd ed). Perth: Department of Health, Western Australia; 2017. Important Disclaimer:

All information and content in this material is provided in good faith by the WA Department of Health, and is based on sources believed to be reliable and accurate at the time of development. The State of Western Australia, the WA Department of Health and their respective officers, employees and agents, do not accept legal liability or responsibility for the material, or any consequences arising from its use. Acknowledgements:

The Chronic Disease Prevention Directorate and the Child Health Promotion Research Centre at Edith Cowan University have developed the Research and Evaluation Framework Implementation Guide with input and advice from a range of individuals and not-for-profit organisations. The Chronic Disease Prevention Directorate is grateful for their thoughtful and constructive comments.

Preface Since 2000, the Department of Health, Western Australia (the Department) has moved away from the direct delivery of statewide health promotion programs to purchasing their delivery through grants and service agreements with a diverse range of not-for-profit organisations (NfPs). In 2010, the responsibility for purchasing these health promotion programs was transferred to the newly-formed Chronic Disease Prevention Directorate (CDPD) within the Public Health Division of the Department. Due to the number of funded NfPs and their variable capacity for research and evaluation, the CDPD identified the need for a research and evaluation framework to inform delivery of and reporting on CDPD-funded health promotion programs. In 2012, the CDPD contracted the Child Health Promotion Research Centre at Edith Cowan University to develop a research and evaluation framework and implementation guide. The guide was intended to support program planning and evaluation while also taking into account best practice approaches and the capacity and needs of NfP and CDPD staff. This work involved a number of activities, including mapping current research and evaluation practices of NfPs; reviewing national and international research and evaluation frameworks and relevant theory-based health promotion planning and evaluation models; consulting with the CDPD, NfPs and external evaluation agencies to examine capacity for research and evaluation and additional support required; holding a forum to present consultation and review findings to stakeholders; and refining the Research and Evaluation Framework and the development of a supporting implementation guide. The first edition of the Research and Evaluation Framework Implementation Guide was released in 2013. In 2016, CDPD consulted with internal policy staff and NfPs using the Research and Evaluation Framework to assess whether the guide was working as intended and to examine whether any improvements could be made. This edition comprises an updated Research and Evaluation Framework Implementation Guide. The guide is intended to be current, relevant and practical, and its content will continue to be developed over time to ensure that it remains so. Denise Sullivan DIRECTOR CHRONIC DISEASE PREVENTION DIRECTORATE

1

Contents Introduction ............................................................................................................. 2 The Research and Evaluation Framework ............................................................ 2 Program Planning Phase ............................................................................. 4 Step 1: Identify the national, state and local context ............................ 5 Step 2: Assess needs, evidence and capacity ..................................... 6 Step 3: Define program goals, objectives and activities ....................... 7 Program Planning Logic Model ............................................................ 8 Research and Evaluation Planning Phase ............................................... 10 Step 4: Develop an evaluation proposal ............................................. 11 Step 5: Complete the evaluation plan ................................................ 12 Evaluation Proposal / Plan ................................................................. 13 Implementation Phase ................................................................................ 16 Step 6: Collect the data ...................................................................... 17 Step 7: Analyse and interpret the data ............................................... 18 Review Phase .............................................................................................. 19 Step 8: Review, recommend and disseminate ................................... 20 Reporting Summary ........................................................................... 21 References ............................................................................................................. 24 Example 1 – Kind Eats Program .......................................................................... 25 Program Planning Logic Model ................................................................. 26 Evaluation Plan ........................................................................................... 27 Reporting Summary ................................................................................... 28 Example 2 – WA Fall Prevention Program .......................................................... 29 Program Planning Logic Model ................................................................. 30 Evaluation Plan ........................................................................................... 31 Reporting Summary ................................................................................... 32 Example 3 – Comprehensive Tobacco Control Program ................................... 33 Program Planning Logic Model ................................................................. 34 Evaluation Plan ........................................................................................... 35 Reporting Summary ................................................................................... 36 Key terms ............................................................................................................... 37 Additional resources ............................................................................................. 38

2

Introduction Research and evaluation are critical components of successful health promotion and a vital step in ensuring that communities benefit from programs being implemented. High quality research and evaluation provide an excellent resource for identifying what is being achieved through a program and its development. Alternatively, when health promotion programs don’t achieve desired effects, research and evaluation help us to understand what went wrong and how it can be improved in future.2 This implementation guide provides a step-by-step process for conducting research and evaluation in the context of health promotion programs using tools, templates and examples. It is important to note that the research and evaluation requirements for different programs will vary widely according to their size and complexity. Therefore, while each step of the Research and Evaluation Framework is relevant to all programs, the nature and focus of evaluation will differ widely from program to program. Consequently, strong partnerships and communication between all stakeholders form a fundamental component of the research and evaluation process.

The Research and Evaluation Framework The Research and Evaluation Framework was informed by various models of health promotion planning and evaluation,3-7 existing research and evaluation frameworks,8-10 and implementation theory.11, 12 The Framework consists of four phases comprising eight steps. The Program Planning phase is designed to help summarise the context in which the program will be implemented (Step 1), to identify program needs, relevant evidence, and capacity for it to be implemented (Step 2), and to define the goals, objectives and activities of the program (Step 3). The Research and Evaluation Planning phase aims to develop a method for assessing whether the program was effective (and why) by first developing an Evaluation Proposal (Step 4), which can be reviewed and developed into a final Evaluation Plan (Step 5). The Implementation phase involves implementing both the program and the research and evaluation plans. Data is collected (Step 6), then analysed and interpreted (Step 7) using methods outlined in the Evaluation Plan. Finally, the Review phase involves reviewing the program, providing recommendations and disseminating findings to relevant stakeholders (Step 8).

3

R & E

Planning Review

Program

Planning

Intended Outcomes

Intended Impacts

Intended Activities

Goals

Objectives

Activities

Outcome Evaluation

Impact Evaluation

Process Evaluation

Context Needs Evidence Capacity

The Problem

Contributing Factors

Response

4

Program Planning Phase

Introduction

Program planning should be informed by national, state and local policy and practice; population needs; evidence from prior interventions; and available capacity. All of these factors will help to inform program goals, objectives and activities and therefore, research and evaluation conducted in light of the program. Aim

In the program planning phase, the aim is to complete a Program Planning Logic Model in order to (1) capture the context in which the program will be implemented, (2) briefly identify the key elements of the program and (3) outline what it is hoped will be achieved through implementing the program. Purpose

The purpose of constructing a logic model is to provide a simple, one-page snapshot of the proposed program. Using a logic model helps put the program in context and identifies the anticipated impacts of specific elements of the program, and how they are expected to contribute to longer-term state or national outcomes.

Templates required:

Program Planning Logic Model

Note that in most cases the Program Planning Logic Model will be a summary of a more detailed program plan.

5

It is important for any health promotion program or service to demonstrate how it links with national, state and local priorities and targets. Step 1 is about recognising the broader picture and significance of the health issue as well as the program’s importance and contribution to reducing the burden of chronic disease and injury. The WA Health Promotion Strategic Framework 2017–2021 is a good place to start, as it details priority areas and strategic directions for:

Curbing the rise in overweight and obesity

Healthy eating

A more active WA

Making smoking history

Reducing harmful levels of alcohol use

Preventing injury and promoting safer communities

Step 1 Tasks

1.1 List the name of the program; agencies involved; time over which the program will run; the overall budget; and the community outcomes in the relevant rows at the top of the Program Planning Logic Model (see page 8). For an idea of what to include, see the examples at the back – starting on page 27).

1.2 Provide a statement in the Program Planning Logic Model under ‘Context’ that justifies the program, by identifying relevant national, state and local strategic plans/policies that relate to the health issue and target group.

Identify the national, state and local context 1

6

Step 2 is about outlining the justification and backing for the program. Identifying the needs of the target population is important in designing the program’s goals and objectives, which in-turn will inform the type of strategies selected. Available evidence and capacity for the program to be implemented will also influence the types of activities chosen. There are many different types of evidence that can be drawn on when deciding what approach to take when designing a health promotion program (e.g., quantitative, qualitative, theory-informed, practice-based, and empirical). If there is minimal evidence or significant gaps in what is known, then formative assessment (such as a needs assessment or a pilot study) may form an initial component of the proposed program.

Step 2 Tasks

2.1 Complete a need for program statement under ‘Context’ that helps justify the program. The statement may include, for example, prevalence of a particular health issue or its contribution to health and/or financial costs.

2.2 Complete an evidence of what works statement under ‘Context’ that helps justify the program activities.

2.3 Complete a capacity to implement statement under ‘Context’ that describes current human, financial, organisational and community resources available to implement the proposed activities. Funding sources should also be listed here.

Assess needs, evidence and capacity 2

7

Step 3 involves describing the activities to be undertaken as part of the program, impacts intended to be achieved by implementing the activities, and the outcomes that the program ultimately hopes to bring about. These outcomes, impacts and activities form the basis of outcome, impact and process evaluation. Program outcomes are the overarching, measurable changes that the program hopes to bring about in the long run. For example, the program may seek to improve adherence to dietary or physical activity guidelines, reduce rates of injury, or increase physical activity in those involved in the program. In most cases, there will be other initiatives working towards the same outcomes and there will be a range of other factors beyond the program that influence progress. Program impacts are short and medium term changes that result directly from the activities delivered as part of the program. These impacts will be quantifiable within the target groups exposed to the program activities. The program is responsible for bringing about these changes. For example, the program may seek to improve awareness or knowledge on a specific topic, such as awareness of the effects of smoking on health, or knowledge of how to correctly calculate body mass index. It is important to ensure that outcomes and impacts are measurable, so they can be evaluated precisely. For example, it is not possible to directly observe increases in confidence, but it is possible to observe increases in scores on a survey designed to assess confidence. Since your program outcomes and impacts will become your goals and objectives, use the SMART acronym when defining your outcomes and impacts (make them Specific, Measurable, Achievable, Realistic and Time-phased).1

Step 3 Tasks

3.1 Consider what the proposed program ultimately intends to achieve for its target population, and describe these long-term outcomes in the logic model under “Program outcomes”.

3.2 Consider the shorter-term impacts required to bring about the program outcomes, and list these in the logic model under ‘Program impacts’.

3.3 Consider program activities that are needed to bring about the program impacts and list these in the logic model under ‘Program activities’. Provide details about each activity including how much, to whom and over what time the activities will be implemented.

Define program outcomes, impacts and activities 3

8

Program Planning Logic Model Steps 1-3: Linking program activities, impacts, outcomes and contextual factors.

Program

Agencies involved

Period (budget)

Community outcomes

Context Program activities Program impacts Program outcomes

What policy / legislation / guidelines are relevant to this program?

Why is this program needed?

What works, according to the evidence?

What resources are available?

What will the program do and who is the target group?

What changes are anticipated as a result of the program activities?

What changes are anticipated as a result of the program impacts?

Formative evaluation Process evaluation Impact evaluation Outcome evaluation

Each column should clearly inform or be informed by adjacent columns Task 1.2

Task 2.1

Task 2.2

Task 2.3

Task 3.1 Task 3.3 Task 3.2

Task 1.1

9

Tips for Defining Impacts and Outcomes

It’s not new, but the SMART1 acronym is a useful way to ensure the evaluation

remains informative. When developing a logic model, make sure the impacts

and outcomes meet the following criteria:

Specific: They should be simple and clear. Make sure they clearly identify

what you want to achieve through the program and with whom.

Measureable: They should be tangible. They need to be written in a way that

allows them to be easily assessed as having been met or not.

Achievable: They should be achievable within the resources and time

available for the program. If impacts and outcomes aren’t possible, it will

simply make the program look like it’s not working.

Realistic: Make sure that the impacts are practicable and that they align with

one or more of the program outcomes.

Time-phased: They should have a time limit on them. Without a time limit,

impacts and outcomes can never be assessed as not having been met.

10

Note that in most cases the Evaluation Plan template will be a summary of a much more comprehensive and detailed evaluation plan.

Research & Evaluation Planning Phase

Introduction

Forward planning is essential to ensure timely collection of high-quality evaluation data. Data collection will likely occur before, during and after the program, not just at the end. Therefore, it is important to know what is required to conduct the evaluation as well as who is involved and when it will occur. Research and evaluation planning assists with this process by outlining program goals, objectives and activities as well as providing information on indicators, data collection and who is responsible for what. Aim

In the research and evaluation planning phase, the aim is to complete an Evaluation Proposal/Plan in order to (1) identify the program goals, objectives and activities, (2) establish a set of indicators, (3) specify whether any additional evaluation questions need answering and (4) indicate how the results of the evaluation and lessons learnt will be disseminated. Purpose

The purpose of constructing an Evaluation Proposal/Plan is to provide a short, simple snapshot of the proposed approach to evaluation. While the level and type of evaluation proposed will depend upon program complexity, duration and maturity, this plan is a summary of the evaluation activities that will occur before, during and after planned program activities. Templates required:

Evaluation Proposal/Plan

11

Step 4 is about developing the Evaluation Proposal that will ultimately, through consultation with relevant stakeholders, become the Evaluation Plan. The proposal links with the Program Planning Logic Model and documents the essential components of the program’s research and evaluation. It provides a snapshot of the entire evaluation process. Once complete, the Evaluation Plan will provide indicators for each goal, objective and activity. In addition, it will summarise where data will be sourced, when it will be collected and who will assume responsibility.

Step 4 Tasks

4.1 List the name of the program; agencies involved; period over which the program will run; budget; and plans for disseminating results in the relevant rows at the top of the Evaluation Plan.

4.2 List the program goal(s) by transferring the ‘Program outcomes’ in the Program Planning Logic Model to the ‘Program goal(s)’ in the Evaluation Proposal.

4.3 List the program objectives by transferring the ‘Program impacts’ in the Program Planning Logic Model to the ‘Program objective(s)’ in the Evaluation Proposal.

4.4 Transfer the ‘Program activities’ from the Program Planning Logic Model into the Evaluation Proposal under ‘Program activities’.

4.5 Specify indicator(s) for each goal, objective and activity that will provide a measure of progress or success in the indicators column.

4.6 For each indicator, describe the source of the data under ‘Source’.

4.7 Enter the dates when the data will be collected and reported under ‘Data collection dates’ and ‘Reporting date(s)’.

4.8 State who will take primary responsibility under ‘Responsibility’.

4.9 List additional questions you wish to answer with the evaluation not already addressed by the existing set of indicators (see page 15 for examples).

Develop an Evaluation Proposal 4

12

Step 5 is about refining the Evaluation Proposal into a full Evaluation Plan. While the majority of thinking about the program and how it will be evaluated has been done prior to this point, this is the time for stakeholders to reach agreement on a final plan, to organise external evaluation expertise (if required) and to conduct formative research to help define strategies or measurement tools. The Evaluation Plan should be reviewed in detail to ensure that the proposed activities are feasible within the budget.

Step 5 Tasks

5.1 Engage stakeholders and review the Evaluation Proposal and budget.

5.2 Finalise the Evaluation Plan.

Complete the Evaluation Plan 5

13

Evaluation Proposal / Plan Steps 4-5: Linking goals, objectives and activities to indicators, data sources, timelines and responsibilities.

Program

Agencies involved

Period (budget)

Planned evaluation outputs

Program goal(s) Outcome indicator(s) Source Data collection dates Reporting date(s) Responsibility

Program objective(s) Impact indicator(s) Source Data collection dates Reporting date(s) Responsibility

Program activities Process indicator(s) Source Data collection dates Reporting date(s) Responsibility

Additional evaluation questions

1.

2.

3.

Task 4.2

Task 4.3

Task 4.5 Task 4.6 Task 4.7 Task 4.8

Task 4.1

Task 4.9

Task 4.4

14

Tips for Developing Meaningful Indicators

Where possible, make them policy relevant

Unless there is a good reason not to, make sure the indicators can be

mapped against relevant policy or guidelines.

Ensure that they accurately assess goals, objectives and activities

It might sound like common sense but indicators often don’t capture exactly

what we want to evaluate, either because we utilise existing data collections

that provide something ‘close enough’, or because our indicators capture only

part of what we’re aiming to evaluate.

Make sure they’re properly operationalised

Often we want to assess changes in concepts we can’t directly measure. We

can’t directly observe increases in knowledge or attitudes but we can observe

changes in measures designed to assess them.

Be mindful of ceiling effects

Indicators need room to move. If we’re looking to demonstrate our program

can increase awareness, then our measures of awareness need room to

improve. Knowing that 97% of respondents were already aware of the health

consequences of a given behaviour prior to participating in a program is

important, but it won’t help you understand whether the program has any

effect.

Don’t be afraid to use qualitative data

Often quantitative indicators are chosen over qualitative ones because we

feel they provide a simpler, clearer answer to our question. However,

qualitative methods will often provide additional useful data, particularly when

evaluating new programs or those with smaller samples.

15

Additional Evaluation Question Examples Beyond assessing whether program goals and objectives have been met and

activities implemented successfully, there will likely be additional questions we

wish to answer as part of an evaluation.

Listed below are a few examples of additional questions we may wish to

answer. The number (and types) of additional questions we wish to address in

an evaluation will be influenced by program complexity and what resources are

available to us.

Has the program been implemented as intended?

What factors impacted on program implementation?

What percentage of the target population did the program reach?

Have demographic factors impacted on program reach?

Were members of the target group satisfied with the program?

Have demographic factors impacted on program effectiveness?

What unanticipated impacts arose from the program?

What were the key barriers to achieving program objectives?

Is the cost of the program justified by the magnitude of the benefits?

Have levels of partnership and collaboration increased?

How could the program be improved?

Are the results consistent with the evidence base?

Is the program sustainable?

Should the program be continued or developed further?

What resources are required to continue or develop the program?

16

Implementation Phase

Introduction

During the implementation phase, evaluation data is collected alongside the implementation of the health promotion program. Analysis of impact data will help to answer questions about the effectiveness of the strategies, while assessment of process data should help to inform why strategies are successful or not. Outcome data will help to provide an indication of progress towards the overall goals of the program. A common cause of concern within the data collection step relates to the ability of staff to obtain accurate data from their sample. Challenges to this process may arise due to a lack of willingness of participants, low literacy among participants and/or participants living in rural or remote areas. Early recognition of potential issues and devising appropriate strategies and data collection tools during the planning phases will help to reduce these barriers. Aim

The aim of the implementation phase is to implement both the Program Plan and Evaluation Plan. Any changes to the implementation of the program or evaluation should be documented.

17

Step 6 involves collecting research and evaluation information according to the methods and timelines outlined in the Evaluation Plan. Collecting accurate and representative data is imperative for assessing the effectiveness of the program. Prior to collecting the data, pilot testing may be required to test whether proposed data collection, storage and analysis methods are feasible. Clearly documenting the data collection process, including difficulties that arise, is an important part of the evaluation. For example, initial response rates, the rate and nature of participant dropout and reported confusion over survey questions will all help to provide context for evaluation results and give an indication of the quality of the data. If external agencies are involved in the collection of data, detailed information about the timing and methods used should be requested.

Step 6 Tasks

6.1 Collect data alongside program implementation as documented in the Evaluation Plan.

6.2 Record process notes regarding any difficulties encountered during data collection that may influence the quality of the data.

Collect the Data 6

18

Step 7 involves conducting the appropriate analyses on the data collected and interpreting the results so that the effectiveness of the program in achieving its intended goals and objectives can be explored. This allows for the strengths and limitations of the program to be identified and for meaningful recommendations to be formulated. As with data collection, clearly detailing how the data is treated and analysed is also an important part of the evaluation. For example, providing details about how the data was prepared for the analysis, and why those avenues were chosen, will help to provide context for the results. It is recommended that a person who is not part of the program implementation team be responsible for the data analysis. This helps to maintain objectivity and to reduce bias in interpreting results. Apart from this, full understanding of the program and discussion with the implementation team is needed to formulate recommendations from the results. If the analysis is being conducted by an external agency, details about the analyses conducted (including justification for it) should be requested.

Step 7 Tasks

6.1 Analyse data as intended in the Evaluation Plan.

6.2 Record process notes regarding how data is treated and analysed (and why) that may impact on its validity and interpretation.

Analyse and Interpret the Data 7

19

Review Phase

Introduction

During the review phase, results are reviewed and recommendations are developed. These are then disseminated to key stakeholders. Formative and process evaluation will provide important guidance around the program’s implementation; impact evaluation will provide evidence of success in achieving the program’s objectives; and outcome evaluation will provide an indication of progress towards the programs ultimate goals. Where possible and appropriate, findings should be disseminated to program partners, community stakeholders, policy makers and the wider health promotion profession. This may take a variety of forms including reports, briefings, seminars, conference presentations, newsletters or peer-reviewed journal publications. This dissemination can contribute to the health promotion evidence base and should be discussed with relevant stakeholders during the reporting process. Aim

The aim of the review phase is to review the findings of the evaluation and to discuss the implications for future program development and sustainable delivery. This phase involves completing the Reporting Summary, which captures the results and challenges of the program and documents recommendations to strengthen future program design and delivery. Purpose

Reviewing findings will help to shape the future of the program and contribute to the evidence base within the health promotion field. It will also contribute to our wider understanding of evidence-based practice and feed back into the first step of the process when proposing ‘innovations’ to the original program. Research and evaluation findings, positive or negative, should be discussed between stakeholders to achieve improvement in the program.

Templates required:

Reporting summary Note that in most cases the Reporting Summary will be a snapshot of a

much more comprehensive and detailed evaluation report.

20

Step 8 involves disseminating the findings and recommendations that have come out of the evaluation. Regardless of the results of an evaluation, understanding why these results transpired can make a valuable contribution to future program development. To provide an example, an early childhood physical activity program may not have produced the desired results due to a limited number of teachers implementing the program. The process evaluation may show a range of barriers for teachers that reduced their capacity to implement the program. This information can then be used to inform future program development by investigating and reducing these barriers prior to further program implementation. This not only contributes to the ‘innovation’ of the program for the future but also its sustainability.

Step 8 Tasks

8.1 List the name of the program, agencies involved, budget and outputs produced (including outputs to be produced) in the relevant rows at the top of the Reporting summary.

8.2 Transfer the program goals, objectives and activities from the Evaluation Plan to the Reporting Summary into the, ‘What did you evaluate?’ column.

8.3 Transfer the outcome, impact and process indicators from the Evaluation Plan to the Reporting Summary into the, ‘How was it measured?’ column.

8.4 Briefly describe the results of the outcome, impact and process evaluation in the appropriate row in the, ‘What did you find?’ column.

8.5 Briefly describe the implications of the results in the appropriate row in the, ‘What are the implications?’ column.

8.6 Briefly describe adaptations made to the Evaluation Plan as well as any implementation challenges that arose throughout the evaluation process in the, ‘What challenges were there?’ column.

8.7 As an overall summary, describe the key findings of the evaluation in terms of program effectiveness, achievements and recommendations in the, ‘Key findings’ row.

Review, Recommend and Disseminate 8

21

Reporting summary Steps 8: Linking the Evaluation Plan to general reporting requirements, recommendations and dissemination

Program

Agencies involved

Budget

Evaluation outputs

Key findings

1.

2.

3.

What did you evaluate? How was it measured? What did you find? What are the implications? What were the challenges?

Program goals

Program objectives

Program activities

Task 8.1

Task 8.2 Task 8.3 Task 8.4

Task 8.7

Task 8.5 Task 8.6

22

Tips for Reporting an Evaluation

Clearly state whether the program is meeting its goals and objectives

The goals and objectives reflect the main reason(s) the program is being

implemented, so be sure to report on whether they’re being met.

Provide implications for the program and for policy

Be sure to state clearly what the implications are for policy and program

development. Involve stakeholders in this process to ensure policy

recommendations are appropriate and any suggested changes to the program are

feasible.

Include stakeholders in the reporting process

Often there are multiple agencies invested in a single evaluation; all with different

needs. Regular contact and updates with those involved will increase the chance

of producing useful outputs for everyone.

Be objective in your assessment of the program

When those evaluating the program are also invested in its implementation, it is

often tempting to highlight successes and overlook failures; however, doing so

means opportunities to further refine and improve the program are lost. Be sure to

highlight successes, but don’t forget to report on areas where objectives aren’t

being met or where activities could be implemented more effectively.

Don’t overstate results

It’s tempting to highlight significant findings, but be careful not to overstate what

the data is telling you. Bear in mind limitations around the sample when reporting

results and be sure to provide confidence intervals or effect sizes to give readers

an idea of how confident they can be in the findings.

23

Suggested Outline for Reporting an Evaluation

Executive Summary

Provide a short, standalone summary of the key points covered by the report.

Introduction

Provide an overview of the program by describing program objectives, target groups,

activities, the agencies involved and the implementation status of the program. Then

describe why the program is needed, including significance of the problem and how

the program addresses a gap in existing services. Finally, define the aims and scope

of the evaluation.

Methodology

Provide a brief overview of the design and methodology. Then describe the sample(s)

and sampling procedure(s). Next describe data collection methods and provide a

timeline that shows when the program was implemented, when data was collected

and the key reporting dates. Finally, identify strengths and limitations of the chosen

methodology and provide justification for it.

Results

Report the findings of the evaluation clearly and objectively against the goals,

objectives and activities outlined in the Evaluation Plan using the agreed indicators.

Results addressing the additional evaluation questions should be presented next.

Discussion

Provide a plain-English summary and interpretation of the evaluation results.

Clearly state the degree to which the objectives of the program and the aims of

the evaluation have been met. Then provide program and policy

recommendations, as well as recommended actions for knowledge translation.

Finally, provide the lessons learned through conducting the evaluation.

Conclusions

Provide a short section that summarises the aims of the evaluation, the key

findings and the recommendations.

24

References 1. Centers for Disease Control and Prevention. Evaluation briefs: writing SMART

objectives. Atlanta, Georgia: Centers for Disease Control and Prevention, 2009.

2. Nutbeam D, Bauman A. Evaluation in a nutshell: a practical guide to the evaluation of health promotion programs. Sydney, NSW: McGraw-Hill; 2010.

3. Bucher J. Using the logic model for planning and evaluation: examples for new users. Home Health Care Management & Practice. 2010;22(5):325-33.

4. Glanz K, Rimer B, Orleans C, Viswanath K. Health behavior and health education: theory, research, and practice. San Fransisco, California: Jossey-Bass; 2008.

5. Glasgow R, Vogt T, Boles S. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American Journal of Public Health. 1999;89:1322-7.

6. Green L, Kreuter M. Health program planning: an educational and ecological approach. 4th ed. New York, NY: McGraw-Hill; 2005.

7. W.K. Kellogg Foundation. Logic model development guide. Battle Creek, Michigan: WK Kellogg Foundation, 2004.

8. Evaluation in health promotion: principles and perspectives. Rootman I, Goodstadt M, Hyndman B, McQueen D, Potvin L, Springett J, et al., editors. Geneva, Switzerland: World Health Organization; 2001.

9. Department of Health, Victoria. Integrated health promotion evaluation planning framework 2010-11 to 2011-12. Melbourne, Victoria: Department of Health, Victoria, 2010.

10. The Health Communication Unit, University of Toronto. Evaluating health promotion programs. Toronto, Ontario: University of Toronto, 2007.

11. Aarons G, Green A, Palinkas L, Self-Brown S, Whitaker D, Lutzker J, et al. Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implementation Science. 2012;7(32).

12. Fixsen D, Naoom S, Blase K, Friedman R, Wallace F. Implementation research: a synthesis of the literature. Tampa, Florida: Louis de la Parte Florida Mental Health Institute, 2005.

25

Example 1

Kindy Eats Program

26

Example 1: Program Planning Logic Model

Program Kindy Eats Program (KEP)

Agencies involved Healthy Kids WA, Department of Health WA

Period (budget) 1 July 2017 – 30 June 2020 ($500,000 per year)

Community outcomes Increased consumption of fruit and veg in WA children; increased proportion of WA children at a healthy weight

Context Program activities Program impacts Program outcomes


WA HPSF 2017-2021 supports programs that improve healthy eating in child care settings.

WHO Global Strategy on Diet, Physical Activity and Health supports programs that help children maintain a healthy diet.


National Health Survey 2014-15 estimates only 16% of 2-3 year olds in WA eat enough fruit and veg.

Australian Health Survey 2011-12 estimates 30% of energy intake in 2-3 year olds is from discretionary foods.

2010 Child Care Centre Survey indicates staff lack confidence, skills and capacity to implement a healthy eating program.


Early intervention is important.

Modelling by parents / carers is important.

Centre policies and staff training are crucial.


Staff FTE: 1.5.

Overall budget of $500,000 / year.

Existing partnerships between parties.


1. Through training, support child care centre staff to implement the KEP.

2. Develop and distribute resources to

centre staff and parents.

3. Contribute to promotional events focussing on child health in WA.


1. Increase (by 50 per year) the number of WA child care centres implementing KEP policies and menus that support healthy eating.

2. With training, increase the percentage of child care centre staff who report a positive attitude toward promoting healthy eating to children.

3. With training, increase the percentage of child care centre staff who report feeling confident with promoting healthy eating to children.


1. Increased fruit and vegetable consumption in children attending participating child care centres.

2. Increased percentage of children attending participating child care centres at a healthy weight.

Formative evaluation Process evaluation Impact evaluation Outcome evaluation

27

Example 1: Evaluation Plan




Planned evaluation outputs 6 month reports, annual reports, evaluation reports, conference presentations, journal articles

Program goal(s) Outcome indicator(s) Source Data collection

dates Reporting date(s) Responsibility

Increase fruit and veg consumption in children attending participating child care centres.

Mean daily serves of fruit and vegetables consumed.

Parent questionnaire. Pre-training and annual follow-ups.

September 30, 2018/2019/2020.

Healthy Kids WA.

Increase percentage of healthy-weight children at participating child care centres.

% of children in healthy BMI range.

Parent questionnaire. Pre-training and annual follow-ups.

September 30, 2018/2019/2020.

Healthy Kids WA.

Program objective(s) Impact indicator(s) Source Data collection


Increase (by 50 per year) the number of WA child care centres implementing KEP policies and menus that support healthy eating.

Number of WA child care centres implementing KEP policies and menus.

KEP training database. Ongoing from Jul 2017 - Jun 2020.

Mar & Sept 30, 2018/2019/2020.

Healthy Kids WA.

With training, increase the percentage of child care centre staff who report a positive attitude toward promoting healthy eating to children.

% of staff with a score of 4 or above on the attitude scale.

Pre-post training KEP questionnaire.

Collected at KEP training.

Mar & Sept 30, 2018/2019/2020.

Healthy Kids WA.

With training, increase the percentage of child care centre staff who report feeling confident with promoting healthy eating to children.

% of staff with a score of 4 or above on the confidence scale.

Pre-post training KEP questionnaire.

Collected at KEP training.

Mar & Sept 30, 2018/2019/2020.

Healthy Kids WA.

Program activities Process indicator(s) Source Data collection


Through training, support child care centre staff to implement the KEP.

Number of centres supported. KEP training database. Ongoing from Jul 2017 - Jun 2020.

Mar & Sept 30, 2018/2019/2020.

Healthy Kids WA.

Mean staff satisfaction score with KEP training.

Post-training KEP questionnaire.

Data collected at KEP training.

Mar & Sept 30, 2018/2019/2020.

Healthy Kids WA.

Develop and distribute resources to centre staff and parents.

Mean satisfaction score with manuals and KEP packs.

Parent questionnaire and post-training KEP questionnaire.

Ongoing from Jul 2017 - Jun 2020.

Mar & Sept 30, 2018/2019/2020.

Healthy Kids WA.

Contribute to promotional events focussing on child health in WA.

Number of promotional events contributed to.

KEP events inventory. Ongoing from Jul 2017 - Jun 2020.

Mar & Sept 30, 2018/2019/2020.

Healthy Kids WA.

Additional Evaluation Questions

1. What factors impacted on program implementation?

2. What were the key barriers to realising program objectives?

3. How could the program be improved?

28

Example 1: Reporting Summary




Evaluation outputs 6 month reports (x3), annual reports (x3), evaluation reports (x1), presentations (x7), journal articles (x3)

Key findings

1. The program led to increases in fruit and veg consumption, but only a modest increase in the percentage of healthy weight.

2. The program brought about substantial improvements in child care centre staff attitudes and confidence with promoting healthy eating to children.

3. Barriers included high staff turn-over within child care centres and low response rates on the parent questionnaire.


Program goals

Increase fruit and veg consumption in children attending participating child care centres.

Mean daily serves of fruit and vegetables consumed.

Moderate increases in mean daily serves of both fruit and veg.

KEP is an effective way to improve fruit and veg consumption in child care settings.

Increase percentage of healthy-weight children at participating child care centres.

% of children in healthy BMI range (measured).

Trivial increase in the percentage of healthy weight children.

Further monitoring is required to examine the impact of KEP on weight in the long-term.

Low response rates on parent questionnaire impacted on statistical power.

Program objectives

Increase (by 50 per year) the number of WA child care centres implementing KEP policies and menus that support healthy eating.

Number of WA child care centres implementing KEP policies and menus.

Excellent uptake of KEP policies and menus.

Methods utilised to make and maintain contact with centres were effective.

With training, increase the percentage of child care centre staff who report a positive attitude toward promoting healthy eating to children.

% of staff with a score of 4 or above on the attitude scale.

Large increase in the % of staff reporting a positive attitude.

KEP is an effective way to improve staff attitudes towards promoting healthy eating to children.

High staff turn-over within participating child care centres.

With training, increase the percentage of child care centre staff who report feeling confident with promoting healthy eating to children.

% of staff with a score of 4 or above on the confidence scale.

Large increase in the % of staff who feel confident.

KEP is an effective way to improve staff confidence with promoting healthy eating to children.

High staff turn-over within participating child care centres.

Program activities

Through training, support child care centre staff to implement the KEP.

Number of centres supported.

Supported 179 centres in total.

Uptake and interest exceeded expectations.

Mean staff satisfaction score with KEP training.

Satisfaction with KEP training was high.

Only minor changes to KEP training were required.

Many staff were unable to attend face-to-face training.

Develop and distribute resources to centre staff and parents.

Mean satisfaction score with manuals and KEP packs.

High satisfaction with KEP resources.

Only minor changes to KEP resources were required.

Costs associated with creating hard-copy resources.

Contribute to promotional events focussing on child health in WA.

Number of promotional events contributed to.

Coordinated: 21 Participated in: 78

Promotional events initially helped generate interest in KEP.

29

Example 2

WA Fall Prevention Program

30


Program WA Fall Prevention Program

Agencies involved Healthy Older Adults WA, Department of Health WA


Community outcomes Reduced fall-related injuries in WA older adults

Context Program Activities Program Impacts Program Outcomes


WA HPSF 2017-2021 supports programs that reduce risk of falls in older adults.

The WA Falls Prevention Model of Care supports falls prevention services targeting older adults in WA.

The National Falls Prevention Plan supports activities that provide best-practice training on falls prevention.


From 2009-2013, 20% of all community injury deaths in WA were caused by a fall.

Falls are the leading cause of death and hospitalisation from injury in people over 65 years.

Nearly half of all injuries in WA adults over 65 years are the result of a fall.


Family and carers have a significant role to play in fall prevention.

A number of studies have shown that improving skills, knowledge and awareness of falls prevention is effective for reducing falls in older adults.


Staff FTE: 2.

Overall budget of $500,000 / year.



1. Design, promote and deliver a series of community workshops on falls and falls prevention for older adults, carers and family members of older adults.

2. Develop and deliver a series of TV and online ads targeting WA adults on the dangers of falls and ways to prevent.


1. Increased knowledge of risk factors contributing to falls in workshop participants.

2. Increased confidence to identify hazards in the home in workshop participants.

3. Increased skills to identify hazards in the home in workshop participants.

4. Increased awareness of factors leading to falls and ways to avoid falls in WA adults exposed to ads.


1. Reduced falls and fall-related injuries in workshop participants.

2. Self-, family- or carer-initiated changes in the home to reduce falls in WA adults by those exposed to ads.

Formative Evaluation Process Evaluation Impact Evaluation Outcome Evaluation

31







Reduce falls and fall-related injuries in older adult workshop participants.

Number of self-reported falls in participating older adults.

Falls diary. Diary provided 3 months prior to training, data collected at pre-training and 3 months post-training.

September 30, 2018/2019.

Healthy Older Adults WA.

Prompt self-, family- or carer-initiated changes in the home to reduce falls in WA adults by those exposed to ads.

Number and type of changes made within the home to prevent falls in adults.

Fall Prevention Survey.

Prior to and following campaign waves.

September 30, 2018/2019.



Increase knowledge of risk factors for falls in workshop participants.

Mean score on the ‘Know the Risks’ quiz.

Workshop questionnaire.

Pre-post training survey and 3-month follow up survey.

Mar & Sept 30, 2018/2019.


Increase confidence to identify hazards in the home in workshop participants.

Mean score on the confidence scale.



Mar & Sept 30, 2018/2019.


Increase skills to identify hazards in the home in workshop participants.

Mean number of hazards identified in ‘Fall Risk Perception Test’.



Mar & Sept 30, 2018/2019.


Increase awareness of factors leading to falls and ways to avoid falls in WA adults exposed to ads.

Mean number of contributing factors and methods to avoid falls identified.



Mar & Sept 30, 2018/2019.



Design, promote and deliver a series of community workshops on falls and falls prevention for older adults, carers and family members of older adults.

% of participants ‘satisfied’ or ‘highly satisfied’ with workshop.


Post-training survey. Mar & Sept 30, 2018/2019.


Number of people attending workshops.

Enrolments database.


Mar & Sept 30, 2018/2019.


Deliver and develop a series of TV and online ads targeting WA adults on the dangers of falls and ways to prevent.

% of surveyed WA adults able to recall content of one or more television or online ads.



Mar & Sept 30, 2018/2019.



1. Have demographic factors impacted on program reach?

2. Have demographic factors impacted on program effectiveness?

3. Is the program sustainable?

32





Evaluation outputs 6 month reports (x2), annual reports (x2), evaluation reports (x1), conference presentations (x3), journal articles (x1)

Key findings

1. The program led to a reduction in both self-reported falls in workshop participants and increased determination to prevent falls in those exposed to advertisements.

2. The program led to large increases in confidence and skills with identifying hazards in the home and small increases in knowledge of risk factors contributing to falls.

3. Improvements to fall rates, confidence and skills achieved at the workshops were all maintained at 3-month follow-up.


Program goals

Reduce falls and fall-related injuries in older adult workshop participants.

Number of self-reported falls in participating older adults.

Moderate reduction in self-reported falls.

The program is an effective way to reduce falls in older adults.

The rate of falls amongst older adults was low, resulting in low statistical power for analysis.

Prompt self-, family- or carer-initiated changes in the home to reduce falls in WA adults by those exposed to ads.

Number and type of changes made within the home to prevent falls in adults.

Small increase in the number of changes made to prevent falls.

The ads only prompted a small increase in self-, family- or carer-initiated changes in the home.

Program objectives

Increase knowledge of risk factors contributing to falls in workshop participants.

Mean score on the ‘Know the Risks’ quiz.

Small increase in knowledge of risk factors.

Sections of the workshop may need to be revised (although see challenges).

Knowledge of risk factors was already high, so ceiling effects may have restricted increases.

Increase confidence to identify hazards in the home in workshop participants.

Mean score on the confidence scale.

Large increase in confidence to identify hazards in the home.

The program is very effective for increasing confidence with identifying hazards.

Increase skills to identify hazards in the home in workshop participants.

Mean number of hazards identified in the ‘Fall Risk Perception Test’.

Large increase in ability to identify hazards.

Program is effective for building skills with identifying hazards.

Increase awareness of factors leading to falls and ways to avoid falls in WA adults exposed to ads.

Mean number of contributing factors and methods to avoid falls identified.

Moderate increases in awareness of falls and ways to avoid falls.

The ads were an effective way to increases awareness about falls and ways to avoid falls.

Program activities

Design, promote and deliver a series of community workshops on falls and falls prevention for older adults, carers and family members of older adults.

% of participants ‘satisfied’ or ‘highly satisfied’ with workshop.

Satisfaction with the workshop was very high.

Workshop content and delivery requires few changes.

Number of people attending workshops.

711 people attended across 41 workshops.

Cost per person was high; more cost-effective ways of delivering workshops should be explored.

Deliver and develop a series of TV and online ads targeting WA adults on the dangers of falls and ways to prevent.

% of surveyed WA adults able to recall content of one or more television or online ads.

40% of adults were able to recall content from at least one TV or online ad.

Reach for the target audience was excellent.

33

Example 3

Comprehensive Tobacco Control Program

34


Program Comprehensive Tobacco Control Program (CTPP)

Agencies involved Healthier Lives WA , Department of Health WA

Period (budget) 1 July 2017 – 30 June 2020 ($1,500,000 per year)

Community outcomes Reduced prevalence of tobacco smoking in WA adults

Context Program Activities Program Impacts Program Outcomes


WA HPSF 2017-2021 supports programs that reduce tobacco smoking in WA adults.

The National Tobacco Strategy 2012-2018 supports programs that reduce the rate of tobacco smoking in Australia.

The WHO Framework Convention on Tobacco Control supports tobacco control measures that reduce the prevalence of tobacco use and exposure.


In 2015, 13% of adults in WA were current smokers.

In 2011, tobacco use was the leading cause of disease burden in Australia.

In 2009/10, tobacco use cost WA $1.26 billion in healthcare costs and lost labour.


A sustained, population-wide, multi-level approach that includes mass media campaigns, access to cessation services, targeted interventions for at-risk groups, community interventions, and tobacco regulation.


Staff FTE: 5.

Overall budget of $1,500,000 / year.



1. Run statewide mass media campaigns targeting WA adults on harms of smoking.

2. Generate community/organisational interest in tobacco control measures.

3. Run PD events to increase knowledge in health professionals throughout the state.

4. Produce / distribute resources to public that support/promote quitting smoking.

5. Run seminars for relevant agencies to raise awareness of harms of second-hand smoking.

6. Provide training on cessation support, treatment services and access pathways for community and health professionals.


1. Increased motivation to quit among smokers exposed to the program.

2. Increased awareness of the harms of smoking and exposure to second hand smoke in adults exposed to the program.

3. Increased attempts to quit smoking in WA smokers exposed to the program.


1. Reduced prevalence of tobacco smoking in WA adults exposed to the program.

Formative Evaluation Process Evaluation Impact Evaluation Outcome Evaluation

35


Program Comprehensive Tobacco Control Program (CTCP)

Agencies involved Healthier Lives WA (HLWA), Department of Health WA




Reduce prevalence of tobacco smoking in WA adults exposed to the program.

% of adults who report smoking daily.

CTCP Survey. Prior to and following campaign waves.

September 30, 2018/2019/2020.

HLWA


Increase motivation to quit among smokers exposed to the program.

% of surveyed smokers ‘highly motivated’ to quit.


Mar & Sept 30, 2018/2019/2020.

HLWA

Increase awareness of the harms of smoking and exposure to second hand smoke in adults exposed to the program.

Mean number of smoking-related health problems recalled by surveyed adults.


Mar & Sept 30, 2018/2019/2020.

HLWA

Increase attempts to quit smoking in WA smokers exposed to the program.

Mean number and duration of self-reported quit attempts.


Mar & Sept 30, 2018/2019/2020.

HLWA


Run statewide mass media campaigns targeting WA adults on harms of smoking.

% of surveyed adults able to recall content from campaign.

CTCP Survey. Post-campaign survey. Mar & Sept 30, 2018/2019/2020.

HLWA

Generate community/organisational interest in tobacco control measures.

% of surveyed adults who recall hearing/seeing quit smoking messages in the past month.

CTCP Survey. Ongoing from Jul 2017 - Jun 2020.

Mar & Sept 30, 2018/2019/2020.

HLWA

Run PD events to increase knowledge in health professionals throughout the state.

Number of attendees at PD events per quarter.

CTCP events and resources database.


Mar & Sept 30, 2018/2019/2020.

HLWA

% of attendees reporting improved knowledge following PD events.

PD/training feedback questionnaire.


Mar & Sept 30, 2018/2019/2020.

HLWA

Produce / distribute resources to public that support/promote quitting smoking.

Number of resources distributed per quarter.



Mar & Sept 30, 2018/2019/2020.

HLWA

Run seminars for relevant agencies to raise awareness of harms of second-hand smoking.

Total seminar attendees per quarter.



Mar & Sept 30, 2018/2019/2020.

HLWA

% of attendees reporting improved awareness following PD events.



Mar & Sept 30, 2018/2019/2020.

HLWA

Provide training on cessation support, treatment services and access pathways for community and health professionals.

% of attendees reporting ‘very good’ awareness following training.



Monthly from Jul 2017 - Jun 2020.

HLWA

Number of health services referring clients to Quitline.

Quitline database. Ongoing from Jul 2017 - Jun 2020.

Monthly from Jul 2017 - Jun 2020.

HLWA

Number of health professionals and others attending training.



Mar & Sept 30, 2018/2019/2020.

HLWA


1. Have demographic factors impacted on program reach?

2. Have demographic factors impacted on changes in attempts to quit smoking?

3. Have partnerships with key stakeholders been strengthened over the course of the program?

36


Program Comprehensive Tobacco Control Program (CTCP)

Agencies involved Healthier Lives WA (HLWA), Department of Health WA


Evaluation outputs 6 month reports (x3), annual reports (x3), evaluation reports (x2), presentations (x19), journal articles (x8)

Key findings

1. The overall prevalence of tobacco smoking decreased amongst WA adults exposed to the program.

2. The program led to increases in motivation to quit, awareness of the harms of smoking and number of quitting attempts in WA smokers exposed to the campaign.

3. The effect of the program on motivation to quit varied by living location and household income.


Program goals

Reduce prevalence of tobacco smoking in WA adults exposed to the program.

% of adults who report smoking daily. Small additional decline in smoking for adults exposed to program.

The program further reduced daily smoking in WA adults.

Program objectives

Increase motivation to quit among smokers exposed to the program.

% of surveyed smokers ‘highly motivated’ to quit.

Large increase in motivation for those exposed to campaign.

The program was effective at increasing motivation to quit.

Effectiveness varied by living location.

Increase awareness of the harms of smoking and exposure to second hand smoke in adults exposed to the program.

Mean number of smoking-related health problems recalled by surveyed adults.

Moderate increase in knowledge of harms of smoking.

The program was effective at increasing awareness of the harms of smoking.

Increase attempts to quit smoking in WA smokers exposed to the program.

Mean number and duration of self-reported quit attempts.

Small increases in the number and length of attempts to quit.

The program was effective at increasing quit attempts.

Program activities

Run statewide mass media campaigns targeting WA adults on harms of smoking.

% of surveyed adults able to recall content from campaign.

60% of adults able to recall content from TV campaign.

Reach for the target audience was excellent.

Reach varied by living location.

Generate community/organisational interest in tobacco control measures.

% of surveyed adults who recall hearing/seeing quit smoking messages in the past month.

Recall of tobacco control messages increased sharply during campaign waves.

Community interest in tobacco control measures was high.

Run PD events to increase knowledge in health professionals throughout the state.

Number of attendees at PD events per quarter.

1407 health professionals in total across 78 PD events.

PD events were successfully delivered.

% of attendees reporting improved knowledge following PD events.

88% of attendees reported improved knowledge.

PD events were very effective at improving knowledge.

Relies on self-reports.

Produce / distribute resources to public that support/promote quitting smoking.

Number of resources distributed per quarter.

1429 resources disseminated. ‘Quit Kits’ were widely disseminated.

Run seminars for relevant agencies to raise awareness of harms of second-hand smoking.

Total seminar attendees per quarter. 322 attendees across 40 seminars. Seminars were effective for raising awareness in key public health agencies.

% of attendees reporting improved awareness following PD events.

79% reported improved awareness following PD events.

Provide training on cessation support, treatment services and access pathways for community and health professionals.

% of attendees reporting ‘very good’ awareness following training.

Increase in number of people reporting ‘very good’ awareness.

Training sessions were a time-effective method for raising awareness amongst staff in key public health agencies and the community.

Regional/remote health professionals more difficult to reach.

Number of health services referring clients to Quitline.

Increase in number of health services referring clients to Quitline.

Number of health professionals and others attending training.

996 people attended training in total across 38 training sessions.

37

Key Terms Community outcome: The underlying reason for implementing a program.

Typically there will be a number of programs all working simultaneously towards the same community outcome. A program may contribute to a community outcome but is not solely responsible for it.

Program outcome: The ultimate, long-term change a program aims to bring about for participants in the program. For example, increases in physical activity or fruit and veg consumption, or reductions in smoking.

Program impact: The intermediary change a program aims to bring about for participants in the program. For example, increases in confidence, skills or knowledge.

Program Goal: An ultimate, long-term aim for a program.

Program Objective: An intermediary aim for a program that, if achieved, should contribute to achievement of one or more program goals.

Program Activity: Action undertaken as part of a program that is intended to contribute to achievement of one or more program objectives.

Formative Evaluation: Evaluation intended to inform program approaches or implementation. It may assess, for example, program need, the policy context, stakeholder views, evidence of what works and available resources.

Process Evaluation: Evaluation intended to examine program activities and how successfully they are being implemented.

Impact Evaluation: Evaluation intended to assess the extent to which program objectives have been met.

Outcome Evaluation: Evaluation intended to examine the extent to which program goals have been met.

Indicator: A measure intended to reflect success with implementing program activities, or progress towards program objectives and goals.

38

Additional Resources Key Health Promotion Evaluation Texts Hawe P, Degeling D, Hall J. Evaluating health promotion: a practitioner’s guide.

Sydney: McLelland and Petty, 1990.

Nutbeam D. The challenges to provide ‘evidence’ in health promotion. Health Promotion International. 1999;14(2):99-101.

Program Planning Chronic Disease Prevention Directorate. WA Health Promotion Strategic

Framework 2017-2021. Perth: Department of Health, Western Australia; 2017.

Bucher JA. Using the logic model for planning and evaluation: examples for new

users. Home Health Care Management & Practice. 2010;22(5):325-333. W.K. Kellogg Foundation. Logic Model Development Guide. Battle Creek,

Michigan: W.K. Kellogg Foundation; 2004.

Renger R, Parker SH, Page M. How using a logic model refined our program to

ensure success. Health Promotion Practice. 2009;10(1):76-82.

Haby M, Bowen S. Making decisions about interventions: a guide for evidence-

informed policy and practice. Melbourne: Department of Health Victoria; 2010.

Research and Evaluation Planning Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health

promotion interventions: the RE-AIM framework. American Journal of Public

Health. 1999;89(9):1322-1327.

Department of Health, Victoria. How to use qualitative research evidence when

making decisions about interventions. Melbourne, Victoria; Department of Health,

Victoria; 2010.

Jolley G, Lawless A, Hurley C. Framework and tools for planning and evaluating

community participation, collaborative partnerships and equity in health

promotion. Health Promotion Journal of Australia. 2008;19(2):152-157.

39

Program and Evaluation Implementation Durlak J, Dupre E. Implementation matters: a review of research on the influence

of implementation on program outcomes and the factors affecting

implementation. American Journal Community Psychology. 2008;41(3-4):327-

350.

Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F. Implementation research: a synthesis of the literature. Tampa, Florida: The National Implementation Research Network, University of South Florida, Louis de al Florida Mental Health Institute; 2005.

Review and Dissemination

Wandersman A, Duffy J, Flaspohler P, Nonan R, Lubell K, Stillman L, et al.

Bridging the gap between prevention research and practice: the interactive

systems framework for dissemination and implementation. American Journal of

Community Psychology. 2008;41(3-4):171-181.

Woolf SH. The meaning of translational research and why it matters. Journal of

the American Medical Association. 2008;299(2):211-213.

Communication notes: reader friendly writing–1:3:25. Ottawa: Canadian Health

Services Research Foundation; 2009.

40

This document can be made available in alternative formats

on request for a person with a disability.

© Department of Health 2017

Copyright to this material is vested in the State of Western Australia unless otherwise indicated. Apart from any fair dealing for the purposes of private study, research, criticism or review, as permitted under the provisions of the Copyright Act 1968, no part may be reproduced or re-used for any purposes whatsoever without written permission of the State of Western Australia.

Date post:	09-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Research and Evaluation Framework Implementation Guide · 2017-04-27 · 2 Introduction Research...

Documents