Program and policy Program and policy evaluationevaluation

PPAS4200PPAS4200

February 1February 1stst, 2012, 2012

Program evaluationProgram evaluation““Systematic collection of information Systematic collection of information about the activities, characteristics, and about the activities, characteristics, and outcomes of programs to make outcomes of programs to make judgements about program, improve judgements about program, improve program effectiveness, and or inform program effectiveness, and or inform decisions about future programming.”decisions about future programming.”

M. Patton. 1997. M. Patton. 1997. Utilization-focused evaluation.Utilization-focused evaluation. Thousand Oaks: Sage, p. 23. Thousand Oaks: Sage, p. 23.

Program evaluation vsProgram evaluation vs Policy evaluation Policy evaluation

Program evaluation: assesses activities that provide services

Policy evaluation: assesses policies, programs, products, individuals (their performance)

Characteristics of evaluationCharacteristics of evaluation

Value focusValue focus Fact-value interdependenceFact-value interdependence Future, present and past orientationFuture, present and past orientation Value dualityValue duality

Source: Dunn, W. 2008. Source: Dunn, W. 2008. Public Policy Analysis. An Introduction.Public Policy Analysis. An Introduction. Upper Saddle River (NJ): Upper Saddle River (NJ): Pearson/Prentice Hall, p. 353.Pearson/Prentice Hall, p. 353.

The planning-evaluation cycle The planning-evaluation cycle

Source: http://www.socialresearchmethods.net/kb/pecycle.htmSource: http://www.socialresearchmethods.net/kb/pecycle.htm

Functions of evaluationFunctions of evaluation

Providing reliable and valid Providing reliable and valid information about policy performanceinformation about policy performance

Contributing to clarification and Contributing to clarification and critique of values underlying goals critique of values underlying goals and objectivesand objectives

May contribute to the application of May contribute to the application of other policy-analytic methods (such other policy-analytic methods (such as problem structuring)as problem structuring)

Source: Dunn, W. 2008. Source: Dunn, W. 2008. Public Policy Analysis. An Introduction.Public Policy Analysis. An Introduction. Upper Saddle River (NJ): Upper Saddle River (NJ): Pearson/Prentice Hall, p. 353-4.Pearson/Prentice Hall, p. 353-4.

Traditional stepsTraditional stepsof policy evaluationof policy evaluation

Specification (what is being Specification (what is being measured)measured)

MeasurementMeasurement AnalysisAnalysis RecommendationRecommendation

Three traditionalThree traditionaltypes of evaluationtypes of evaluation

Impact evaluationImpact evaluation Process evaluationProcess evaluation Efficiency evaluationEfficiency evaluation

– Cost/benefit analysisCost/benefit analysis

Approach to program evaluationDeveloped by the City of Ottawa

Source: http://ottawa.ca/en/city_hall/funding/toolkit/

Four functions of monitoringFour functions of monitoringAdapted from Adapted from www.policyonline.org – chap. 6 – chap. 6

Compliance of actionsCompliance of actions

With legislative procedures, regulatory With legislative procedures, regulatory agencies, professional bodies, etc.agencies, professional bodies, etc.

Resources and services reaching Resources and services reaching target groupstarget groups

Ex.: are revenues intended for local Ex.: are revenues intended for local communities reaching them?communities reaching them?

Four functions of monitoringFour functions of monitoringAdapted from Adapted from www.policyonline.org – chap. 6 – chap. 6

AccountingAccounting

Measuring economic and social changesMeasuring economic and social changes

Ex.: Ex.: EU education and training policies

ExplanationExplanation

Why outcomes may differWhy outcomes may differ

The professionalisationThe professionalisationof policy evaluation – Prosof policy evaluation – Pros

Adapted from: Jacob, S., and Y. Boisvert. 2010. “To Be or Not to Be a Profession: Pros, Cons and Challenges for Evaluation.” Adapted from: Jacob, S., and Y. Boisvert. 2010. “To Be or Not to Be a Profession: Pros, Cons and Challenges for Evaluation.” EvaluationEvaluation 16 (4): 349-369. 16 (4): 349-369.

Credibility and legitimacy of the fieldCredibility and legitimacy of the field Increasing training offeringsIncreasing training offerings Enhancing and improving the status Enhancing and improving the status

and prestige of evaluationand prestige of evaluation

The professionalisationThe professionalisationof policy evaluation – Prosof policy evaluation – Pros

Adapted from: Jacob, S., and Y. Boisvert. 2010. “To Be or Not to Be a Profession: Pros, Cons and Challenges for Evaluation.” Adapted from: Jacob, S., and Y. Boisvert. 2010. “To Be or Not to Be a Profession: Pros, Cons and Challenges for Evaluation.”

EvaluationEvaluation 16 (4): 349-369 16 (4): 349-369

Facilitating the selection of Facilitating the selection of evaluators and improving the quality evaluators and improving the quality of evaluationsof evaluations

Protecting the publicProtecting the public Avoiding problematic/unethical Avoiding problematic/unethical

behaviorbehavior

The professionalisationThe professionalisationof policy evaluation – Consof policy evaluation – Cons

Adapted from: Jacob, S., and Y. Boisvert. 2010. “To Be or Not to Be a Profession: Pros, Cons and Challenges for Evaluation.” Adapted from: Jacob, S., and Y. Boisvert. 2010. “To Be or Not to Be a Profession: Pros, Cons and Challenges for Evaluation.” EvaluationEvaluation 16 (4): 349-369. 16 (4): 349-369.

Homogenizing evaluation and Homogenizing evaluation and restricting diversityrestricting diversity

Reducing training offeringsReducing training offerings Restricting or blocking access to the Restricting or blocking access to the

professionprofession