08 Monitoring Policy Outcomes

8/4/2019 08 Monitoring Policy Outcomes

1/47

11

MONITORING POLICY OUTCOMES

Discussant: Samuel O. Parami, MM., MPD.

DDA 326: Public Policy Analysis


2/47

2 2

Presentation Outline

A. IntroductionB. Monitoring in Policy Analysis

1. Sources of Information2. Types of Policy Outcomes

3. Types of Policy Actions4. Definitions and Indicators

C. Approaches to Monitoring1. Social Systems Accounting2. Social Experimentation3. Social Auditing4. Research & Practice Synthesis

D. Techniques for Monitoring


3/47

3 3

The Power of Measuring Results

If you do not measure results, you can not tellsuccess from failure

If you can not see success, you can notreward it

If you can not reward success, you are probablyrewarding failure

If you can not see success, you can not learnfrom it

If you can not recognize failure, you can notcorrect it

If you can demonstrate results, you can winpublic support

Adapted from Osborne & Gaebler, 1992


4/47

4 4

Introduction

The consequences of policy actions are neverfully known in advance and, for this reason, it

is essential to monitor policy actions after theyhave occurred. In fact, policy recommendations may be

viewed as hypotheses about the relationshipbetween policy actions and policy outcomes:if action A is taken at time t 1, outcome O willresult at time t 2 .


5/47

5 5

A. Monitoring in Policy Analysis Monitoring is the policy-analytic procedure

used to produce information about the causesand consequences of public policies.

Monitoring, since it permits analysts todescribe relationships between policy-program operations and their outcomes, is theprimary source of knowledge about policy

implementation. Monitoring is primarily concerned about with

establishing factual premises about public

policy.


6/47

6 6

A. Monitoring in Policy Analysis Monitoring performs at least four major

functions:

1. Compliance: monitoring helps determine

whether the actions of programadministrators, staff, and other stakeholdersare in compliance with standards andprocedures imposed by legislatures,regulatory agencies, and professional bodies.

2. Auditing: monitoring helps determine whetherresources and services intended for certaintarget groups and beneficiaries have actuallyreached them.


7/477 7

A. Monitoring in Policy Analysis Monitoring performs at least four major

functions:

3. Accounting: monitoring produces

information that is helpful in accountingfor social and economic changes thatfollow the implementation of broad sets ofpublic policies and programs over time.

4. Explanation: monitoring also yieldsinformation that helps to explain why theoutcomes of public polices and programsdiffer.


8/478 8

A. Monitoring in Policy Analysis

1. Sources of information . Information must be:

a. Relevant. Macronegative versus micropositive.

b. Reliable.

Observations precise and dependable.c. Valid.

Information about policy outcomes shouldmeasure what we think it is measuring.


9/479 9


1. Sources of information . Information on policy outcomes.

Historical Statistics of theUnited States

Current Opinion

Statistical Abstract of theUnited States United States Census of Population byStates

County and City Data Book Congressional District Data Book

Census Use Study National Opinion Research CenterGeneral Social Survey

Social and EconomicCharacteristics of Students

Survey Research Center NationalElection Studies

Educational Attainment inthe United States

National Clearinghouse for MentalHealth Information

Current Population Reports National Clearinghouse for Drug Abuse

Information


10/4710 10


1. Sources of information . Information on policy outcomes.

Current Population Reports National Clearinghouse for Drug AbuseInformation

The Social and EconomicStatus of the BlackPopulation in the UnitedStates

National Criminal Justice ReferenceService

Female Family Heads Child Abuse and Neglect ClearinghouseProject

Monthly Labor Review National Clearinghouse on RevenueSharing

Handbook of Labor Statistics Social Indicators

Congressional Quarterly State Economic and Social Indicators


11/471111


1. Sources of information .

When information is not available fromexisting sources, monitoring may becarried out by some combination of

questionnaires, interviewing, fieldobservation, and the use of agencyrecords.


12/4712 12


2. Types of policy outcomes.a. Consequences:

Policy outputs the goods, services, orresources received by target groups andbeneficiaries.

Policy impacts actual changes in behaviorand attitudes that result from policy outputs.

b. Populations: Target groups individuals, communities, ororganizations on whom policies and programsare designed to have an effect.

Beneficiaries

groups for whom the effects ofolicies are beneficial or valuable.


13/4713 13


3. Types of policy actions. Policy actions havetwo major purposes.

a. Regulation actions designed to ensurecompliance with certain standards orprocedures.

b. Allocation actions that require inputs ofmoney, time, personnel, and equipment.

Regulative and allocative actions may haveconsequences that are distributive orredistributive.


14/4714 14


4. Types of policy actions.a. Policy inputs the resources time,

money, personnel, equipment, andsupplies used to produce outputsand impacts.

b. Policy processes the

administrative, organizational, andpolitical activities and attitudes thatshape the transformation of policyinputs into policy outputs andimpacts.


15/4715 15


ISSUEAREA

POLICY ACTIONS POLICY OUTCOMES

Inputs Processes Outputs Impacts

CriminalJustice.

Expendituresfor salaries,equipment,maintenance.

Illegal arrestsas percentagesof total arrests.

Criminalsarrested per100,000known crimes.

Criminalsconvicted per100,000known crimes.

MunicipalServices.

Expendituresfor sanitation

workers andequipment.

Morale amongworkers.

Totalresidences

served.

Cleanliness of streets.

Social welfare. Number of social workers.

Rapport withwelfarerecipients.

Welfare casesper socialworker.

Standard of living of dependentchildren.


16/4716 16


4. Definitions and indicators. Variables versus constants. Definitions.

Constitutive definitions

gives meaning towords used to describe variables by usingsynonymous words.

Provide no concrete rules or guidelines foractually monitoring changes.

Operational definitions gives meaning to avariable by specifying the operations necessaryto experience and measure it.

Indicators directly observable characteristics.


17/4717 17

B. Approaches to Monitoring

APPROACHTYPES OFCONTROL

TYPE OFINFORMATIONREQUIRED

Social systems

accountingQuantitative

Available and/or new

information

Social experimentationDirect manipulationsand quantitative

New information

Social auditingQuantitative and/orqualitative New information

Research and practicesynthesis

Quantitative and/orqualitative

Available information


18/47

18 18

B. Approaches to Monitoring

POLICY POLICY POLICY POLICYINPUTS PROCESSES OUTPUTS IMPACTS

PRECONDITIONS UNFORESEEN SIDE-EFFECTSEVENTS AND

SPILLOVERS

CONTROLLED OUTCOMESMANIPULABLE ACTIONS

UNMANIPULABLE CAUSES UNCONTROLLED EFFECTS

In1In2...Ini

P 1P 2...P j

O 1O 2...O m

Im 1Im 2...Im n

PC 1PC 2...

PC p

E 1E 2...

E q

SES 1SES 2...

SES r


19/47

19 19

1. Social Systems Accounting

An approach and set of methods that permit

analysts to monitor changes in objective and

subjective social conditions over time.


20/47

20 20


The major analytic element of socialsystems accounting is the social indicator .

Statistics that measure socialconditions and changes therein overtime for various segments of apopulation. By social conditions, wemean both the external (social andphysical) and the internal (subjectiveand perceptional) contexts of human

existence in a given society.

1 S i l i


21/47

2121

1. Social systems accounting

AREA INDICATORHealth and illness Persons in state mental hospitals

Public safety Persons afraid to walk alone at night

Education High school graduates aged 25 and older

Employment Labor force participation by women

Income Percent of population below the poverty lineHousing Households living in substandard units

Leisure and recreation Average annual paid vacation inmanufacturing

Population Actual and projected population

Government and politics Quality of public administration

Social values andattitudes

Overall life satisfaction and alienation

Social mobility Change from fathers occupation

Physical environment Air pollution index

Science and technology Scientific discoveries

Representative social indicators


22/47

22 22


Must assume that the reasons forindicator change are related to policyactions. This is not always the case.

Advantages. Identify areas of insufficient information.

When indicators provide reliableinformation, possible to modify policies.


23/47

23 23


Limitations. Choice of indicators influenced by values

of analysts. Social indicators frequently do not cover

manipulable policy actions. Most social indicators are based available

data about objective (rather than

subjective) social conditions. Social indicators provide little information

about how inputs are transformed intooutcomes.


24/47

24 24



25/47

25 25



26/47

26 26

Coefficients a

13.516 2.561 5.277 .000

.340 .243 .322 1.400 .17971.419 11.060 6.458 .000-1.551 .403 -.705 -3.849 .002-6.745 22.133 -.305 .7641.218 .494 .525 2.465 .025

(Constant)

Number of years(Constant)Number of years(Constant)Number of years

Model1

1

1

Breaks indisaster patterns1950 - 1971

1972 - 1988

1989 - 2006

B Std. Error

UnstandardizedCoefficients

Beta

StandardizedCoefficients

t Sig.

Dependent Variable: Total Disaster Declarationsa.

Model Summarya

Breaks in disaster patterns R R Square Sig.1950 - 1971 0.322 0.1031972 - 1988 0.705 0.497 ***1989 - 2006 0.525 0.275 *aPredictors: (Constant), Number of years

*p


27/47

27 27



28/47

28 28

2. Social Experimentation Use of social indicators often leads to random

innovation rather than systematic change.

Social experimentation is the process ofsystematically manipulating policy actions inway that permits more or less preciseanswers to questions about the sources ofchange in policy outcomes.


29/47

29 29

2. Social Experimentation

Characteristics and procedures. Direct control of experimental treatment. Comparison (control) groups. Random assignment.

Goal: internal validity the capacity ofexperiments and quasi-experiments tomake valid causal inferences about theeffects of actions on outcomes.


30/47

30 30


Threats to internal validity. History. Maturation. Instability. Instrumentation and testing.

Mortality. Selection. Regression artifacts.


31/47

3131


Social experimentation is weakestin the area of external validity orgeneralizability.

Neglects policy processes.


32/47

32 32

3. Social Auditing

One of the limitations of social systemsaccounting and social experimentationis that both approaches neglect oroversimplify policy processes.


33/47

33 33

3. Social Auditing

Social auditing explicitly monitorsrelations among inputs, processes,outputs, and impacts in an effort to trace

policy inputs from disbursement to finalrecipient. Social auditing helps to determine

whether ineffective outcomes are theresult of inadequate inputs or processesthat divert resources or services from

intended target groups or beneficiaries.


34/47

34 34

3. Social Auditing

Processes monitored of twomain types:

Resource diversion (from targetor beneficiary groups).

Resource transformation(changes in meaning of policyactions from administrator torecipient).


35/47

35 35

Social Audit Steps

The three phases of a social audit

Phase 1: Design and data collection

a. Clarify the strategic focusb. Design instruments, pilot test

c. Collect information from householdsand key informants in a panel ofrepresentative communities


36/47

36 36

Social Audit Steps

Phase 2: Evidence-based dialogue and

analysisa. Link household data with information from

public services

b. Analyze findings in a way that points toaction

c. Take findings back to the communities for

their views about how to improve thesituation

d. Bring community members into discussionof evidence with serviceproviders/planners.


37/47

37 37

Social Audit Steps

Phase 3: Socialization of evidence forpublic accountability

Work-shopping

Communication strategy Evidence-based training of planners

and service providers Media training Partnerships with civil society


38/47

38 38

4. Research and practice synthesis.

An approach to monitoring that involves thesystematic compilation, comparison, andassessment of the results of past efforts toimplement public policies.

Two primary sources of available information.a. Case studies of policy formulation and

implementation.

b. Research reports that address relationsamong policy actions and outcomes.


39/47

39 39

4. Research and practice synthesis.

Two methods.a. Case survey method a set of

procedures used to identify andanalyze factors that account forvariations in the adoption andimplementation of policies.

Requires case coding scheme, a set ofcategories that capture key aspects ofpolicy inputs, processes, outputs, andimpacts.


40/47

40 40

Research and practice synthesis

Two methods.b. Research survey method a set of

procedures used to compare andappraise results of past research onpolicy actions and outcomes. Yields empirical generalizations about sources

of variation in policy outcomes, summaryassessments of the confidence researchershave in these generalizations, and policyalternatives or action guidelines that are impliedin these generalizations.

Requires the construction of a format forextracting information about research reports.


41/47

4141

4. Research and practice synthesis

Advantages.

a. Comparatively efficient way to compile andappraise an increasingly large body of cases andresearch reports on policy implementation.

b. The case survey method is one among severalways to uncover different dimensions of policyprocesses that affect policy outcomes.

c. The case survey method is a good way toexamine both objective and subjective conditions.


42/47

42 42

4. Research and practice synthesis

Limitations. All related to reliability and validity of

information.


43/47

43 43

4. Research and Practice Synthesis

Example

David H. Greenberg; Charles Michalopoulos; Philip K. Robins.

Do Experimental and NonexperimentalEvaluations Give Different Answers about theEffectiveness of Government-Funded TrainingPrograms?


44/47

44 44

4. Research and Practice Synthesis: Exampl e

Abstract

This paper uses meta-analysis to investigatewhether random assignment (or experimental)

evaluations of voluntary government-fundedtraining programs for the disadvantaged haveproduced different conclusions thannonexperimental evaluations.

Information includes several hundredestimates from 31 evaluations of 15 programsthat operated between 1964 and 1998.


45/47

45 45

4. Research and Practice Synthesis: Example

The results suggest that experimentaland nonexperimental evaluations yieldsimilar conclusions about theeffectiveness of training programs, butthat estimates of average effects for

youth and possibly men might havebeen larger in experimental studies.


46/47

46 46

4. Research and Practice Synthesis: Example

The results also suggest that variationamong nonexprimental estimates ofprogram effects is similar to variationamong experimental estimates for menand youth, but not for women (for whomit seems to be larger), although small

sample sizes make the estimateddifferences somewhat imprecise for allthree groups.


47/47

C. Techniques for Monitoring

APPROACHGRAPHICDISPLAYS

TABULARDISPLAYS

INDEXNUMBERS

INTER-RUPTEDTIME-SERIESANALYSIS

CONTROL-SERIESANALYSIS

REGRES-SION DIS-CONTI-NUITYANALYSIS

Socialsystemsaccounting X X X X X O

Socialauditing x x x x x oSocialexperimen-

tation x x x x x xResearchandpracticesynthesis x x o o o o

Date post:	07-Apr-2018
Category:	Documents
Upload:	nilo-lagura
View:	216 times
Download:	1 times

08 Monitoring Policy Outcomes

Documents