Date post: | 27-Dec-2015 |
Category: |
Documents |
Upload: | avice-ferguson |
View: | 215 times |
Download: | 2 times |
Unit 10Evaluation and Policy Change
• Check your devices and make sure you can see & hear
• Have your syllabus handy in the event you have any questions
• You can always contact me by email, schedule an appointment for office hours or schedule a telephone conference
Unit 9 Evaluation and Policy Change
1
Unit 9 Evaluation and Policy Change
4
Overview: Evaluation and Policy Change
• Introduction• Problems in evaluating public programs
– goal specification and goal change– measurement– targets– efficiency and effectiveness– values and evaluation– politics– increasing requirements for evaluation
• Policy change
Unit 9 Evaluation and Policy Change
5
Introduction
• The final stage of the policy process is to assess what has occurred as a result of policy selection and implementation and to make appropriate policy changes:– Evaluating policies is a difficult and highly political process. – Policy change is also difficult, as public-sector organizations
have many methods that they can use to resist policy change.
• We should not assume that government organizations are always resistant to change:– Most organizations resist change to some degree, but most
organizations want to correct their weaknesses.– Many of the obstacles to public-sector change are external to
public organizations—stemming from Congress and the public.
Unit 9 Evaluation and Policy Change
6
Problems in Evaluating Public Programs
• Evaluating public programs involves:– cataloging the goals of the program– measuring the extent to which goals are
achieved– suggesting changes that may bring
performance more in line with the goals of the program
– difficulty of producing unambiguous measurements of performance in a public organization
Unit 9 Evaluation and Policy Change
7
Goal Specification and Goal Change
• The legislation establishing a program should be a source of goal statements. Remember that:– Legislation is frequently written in vague language.– It is often difficult to attach readily quantifiable goals.– Goals specified in legislation may be impossible or
contradictory.– Internal political dynamics can become particularly
important in the absence of clear goals.– Contradictions in goals may also exist across
organizations in government (tobacco subsidies versus antismoking campaigns).
Unit 9 Evaluation and Policy Change
8
Goal Specification and Goal Change
• Internal political dynamics may modify existing goals:– Roles can be expanded in positive ways:
• The Army Corps of Engineers transformed its image from one of environmental disregard to one of environmental sensitivity.
– Goal changes can also be negative, as with the capture of regulatory bodies by regulated industries.
– The “displacement of goals” by members of an organization is more common than capture:
• Individuals in the organization may become focused on personal survival and aggrandizement rather than the mission of the organization.
• The goals of the organization may become more focused on maintenance and survival than on its mission.
• Anthony Downs describes the “life cycle” of organizations, beginning as advocates of social causes but over time emphasizing on personal survival and the maintenance of budgets.
• In these instances, the operating goals of the program deteriorate.
Unit 9 Evaluation and Policy Change
9
Goal Specification and Goal Change
• There are several types of goal change:– displacement: individuals exhibit transformation; reflexive transformation– empire building: organizations exhibit transformation; reflexive
transformation– street-level bureaucracy: individuals exhibit transformation; operational
transformation– adaptation: organizations exhibit transformation; operational
transformation• When transformations are exhibited individually, the focus of
evaluation should be correcting individual behavior, whereas organizational transformations should call evaluative attention to the policies of the organization.
• Efforts to minimize or avoid goal displacement include:– making managers more directly responsible for organizational
performance– instituting performance pay schemes at lower levels– making the public sector more consumer driven
Unit 9 Evaluation and Policy Change
10
Goal Specification and Goal Change
• Goals may be impractical:– The Preamble to Constitution expresses many goals but does so
in vague language.– The Employment Act of 1946 pledges the government to
maintain “full employment,” but the government plays the “numbers game” to attempt to prove that the goals have been reached.
• Most public organizations service multiple constituencies:– The Comprehensive Employment and Training Act (CETA)
performed different functions for different groups in society.– The process of evaluation requires that analysts ask whose
goals are being achieved.• Goals may be either straitjackets or opportunities—the
organization is told both what to do and what not to do.
Unit 9 Evaluation and Policy Change
11
Measurement
• The efficiency and effectiveness of government is limited by the absence of a ready means to judge the value of what is being produced.
• Activity measures are often substituted for output measures in order to evaluate public sector performance; this serves the interests of existing organizations:– It shields organizations from stringent evaluations on
nonprocedural criteria.– Action becomes equated with success.– It may also have the less obvious effect of keeping clients
attached to a program even when they no longer need the program.
Unit 9 Evaluation and Policy Change
12
Factors that Inhibit Adequate Measurement
• Several factors inhibit adequate measurement of government performance. These are:– The time-span over which benefits of programs occur:
• Lester Salamon’s analysis of “sleeper” effects of the New Deal programs: initially viewed as failures but significant results were apparent thirty years after the programs were terminated.
• the “flip side” of the time problem:– the effects of a program should be durable– example: Head Start does not maintain high performance without
reinforcement in later years, reflecting the effects of program decay
• the time element may produce political difficulties– individuals have little time, and must produce results quickly (especially
elected officials)– policy process to focus on short-term successes– bureaucrats may observe a more long-term perspective
Unit 9 Evaluation and Policy Change
13
Factors that Inhibit Adequate Measurement
• Additional factors affecting the population:– It is difficult to isolate the effects of different programs.– It is difficult to hold constant all social and economic
factors other than the one being examined.• The history of the program and the individuals
involved:– Existing programs may jeopardize the success of a
new program.– Clients and administrators may become cynical about
the likely success of a program.– All policy areas seem to be confounded by almost
constant cycles of change and contradiction.
Unit 9 Evaluation and Policy Change
14
Factors that Inhibit Adequate Measurement
• An organizational basis of evaluation, which may limit the scope of the inquiry– Different organizations will measure different aspects
of a program.– A given agency may not evaluate the unintended
effects of a program.– For example, interstate highway system projects are
evaluated differently by engineers and mayors:• Engineers probably regard the highways to be a success in
terms of saved lives and fuel, whereas • Mayors of large cities may see the highways as a failure
because they facilitated “white flight” and reduced city tax bases.
Unit 9 Evaluation and Policy Change
15
Factors that Inhibit Adequate Measurement
• Reactive effects of experimental evaluation, which may influence results:– People may alter their behavior other under the
conditions of a policy experiment.– Individuals may work to produce results from
programs that favor their interests.– If an experimental method is not used, unmeasured
social and economic factors, not the program, may be the cause of any observed effects on the target population.
– Problems of research design reduce analysts’ ability to make definitive statements about the real worth of a policy.
Unit 9 Evaluation and Policy Change
16
Targets• Evaluators need to know not only the goal of the program but also
whom it is intended to affect:– Medicare—although the health of the elderly population has improved in
general, the health of the neediest portion of the elderly population has not commensurately improved
– Head Start—has not been used by the poorest families because it is a part-day program
• Participation in many programs is voluntary, and this may create problems for organizations seeking to serve particular target audiences.
• A program may create a false sense of success by “creaming” the segment of the population served by a program:– Programs may select clients who need little help in order to report more
success, and this creates a problem of generalizing effects for an entire population.
• Selecting target audiences is inherently political, as programs may seek to broaden audiences to build political support:– Broadened audiences make program evaluation more difficult.
Unit 9 Evaluation and Policy Change
17
Efficiency and Effectiveness• Evaluating efficiency requires assessing the ratio of the
cost of the efforts to the results:– In measuring efficiency, it is difficult to determine results and
assign costs.• Similar problems arise in measuring effectiveness
because surrogate measures of intended results are often required.
• Due to these problems, much of the assessment of performance in government depends on the evaluation of procedural efficiency:– Proceduralism may cause goals to be displaced, shifting the
focus of evaluation to the process itself rather than the services that the process is intended to produce.
– Concern with measuring efficiency through procedures may actually reduce the efficiency of the policy process because procedural steps and red tape exacerbate service delivery problems.
Unit 9 Evaluation and Policy Change
18
Values and Evaluation• The analyst who performs an evaluation requires a value system to
enable him or her to assign valuations to outcomes:– Value systems are by no means constant across the population or
across time.• There is difficulty in determining the proper valuation and weighting
of the outcomes of a program, as values vary across persons and over time:– This problem may be particularly relevant when a program has
significant unintended effects.• The analyst brings his or her own values to the evaluation process:
– Such values may be manifested through the policy process.– They may also have substantial impact on the final evaluation of
outcomes.• There are also other sources of values:
– organizations and professions, and – these values may affect the policy process
• Values may become the real battleground while rational argumentation and policy analysis are merely the ammunition.
Unit 9 Evaluation and Policy Change
19
Political Context
• Evaluations of public programs are performed in a political context.
• Political leaders may be interested only in those benefits created for their constituents and may disregard overall program effectiveness.
• A program may be evaluated only to validate decisions that have already been made—evaluators are often called upon to do “quick and dirty” evaluations.
• Institutional evaluators are therefore important because they are relatively impartial and stable in their evaluations.
Unit 9 Evaluation and Policy Change
20
Increasing Requirements for Evaluation
• There is a more recent focus on outputs (effects) rather than inputs (budget and staff) of the government.
• Congress passed the 1993 Government Performance and Results Act (GPRA), with the basic idea of appraising government organizations on the basis of their strategic plans and on quantitative measures:– The danger is that Congress will focus on only a few simple quantitative
measures and fail to understand the complexities involved.– The Gore Report was similar to GPRA in its emphasis on evaluation.– Continuing regulatory reviews have also required additional economic
evaluations, even if the evaluations have at times been a bit superficial.– Educational quality is increasingly subject to evaluation, although it is
still being measured primarily by standardized tests.– The George W. Bush’s administration’s focus on effort to punish poorly
performing schools are based on rather simplistic measures of progress; creating additional risks.
Unit 9 Evaluation and Policy Change
21
Policy Change
• After evaluation, the next stage in the process is policy change.
• Rarely are policies maintained in exactly the same form over time; instead, they are constantly evolving:– Change is often the result of:
• changes in the socioeconomic or political environment;• learning on the part of the personnel administering the program; or• simple elaboration of existing structures or ideas
• Policymaking in industrialized countries is often the result of policy change rather than the rise of new issues:– Most policy areas are already populated by a number of
programs and policies.• Policy succession—the replacement of one policy by
another—is therefore an important concept now.
Unit 9 Evaluation and Policy Change
22
Outcomes of Policy Evaluation
• Policy Maintenance: occurs only rarely as a matter of choice—it may result from a failure to make decisions:– This is an unlikely outcome of evaluation,
however, because politicians make “names” for themselves by advocating new legislation, not maintenance.
– Original policy decisions are generally flawed in some way, so it is likely that potential improvements will be identified.
Unit 9 Evaluation and Policy Change
23
Outcomes of Policy Evaluation
• Policy termination is also unlikely. – Programs acquire a life of their own—they develop
organizations, hire personnel, and develop a clientele.– Programs that build a “stock” of expected future benefits are
particularly difficult to terminate:• Social Security created a “stock” of future benefits for its clientele. A
reduction in benefits would create hardships for its clientele.• Welfare and food stamps do not involve planning by recipients, so
there is no “stock” element; makes it easier to reduce or terminate programs.
– The Reagan administration was unable to make sweeping program reductions. The George H.W. Bush administration wanted to eliminate the Department of Education but found that promise difficult to keep.
– While public programs are not immortal, relatively few are fully terminated.
Unit 9 Evaluation and Policy Change
24
Outcomes of Policy Evaluation• Policy succession is the most likely outcome of an existing program.• There are several forms of policy succession:
– linear: the direct replacement of one program or policy by another, or the simple change of location of an existing program:
• Example: the replacement of the Aid to Families with Dependent Children welfare program by the Personal Responsibility and Work Opportunity Reconciliation Act of 1996.
– consolidation: placing several programs that have existed independently into a single program:
• Example: rolling together a number of categorical health and welfare programs into a few block grants during the Reagan era.
– splitting: dividing programs into two or more individual components in a succession:
• Example: the 1974 split of the Atomic Energy Commission into the Nuclear Regulatory Agency and the Energy Research Development Agency.
– nonlinear: complex successions of policies and organizations, involving elements of other types of successions
• Example: the multiple changes involved in the creation of the Department of Energy.
Unit 9 Evaluation and Policy Change
25
Policy Succession• Policy Succession: The process is somewhat distinctive even
though it may entail processes like the following:• Agenda setting; not as difficult as in policy initiation stage:
– The broad issue has already been accepted as an agenda item and is now returned to a particular institutional agenda.
• Legitimation and formulation:– Client and producer interests are threatened by many proposed policy
changes, especially in cases of likely policy consolidation.– Existing clients are likely to fight to maintain the status quo in many
cases.• Some policy successions are internally generated:
– External political forces may help effectuate, and even help sponsor, internally generated policy change.
– Some program managers are risk takers and are willing to bet on the benefits of policy change.
– Some organizations will gladly shrink the size of overextended programs such that organizational “heartlands” are preserved.
Unit 9 Evaluation and Policy Change
26
Policy Succession• Clientele groups may seek to split a program from a larger
organization in order to develop a clearer target for their political activities.
• Coalitions for policy change:– Forming a coalition requires attention to the commitments of individual
members of Congress to particular interests and ongoing programs.– It requires use of mechanisms of partisan analysis, logrolling, and pork-
barrel legislation.– Building broadly interested coalitions for policy change may threaten the
existence of the policy, as termination may be the only alternative on which the coalition can agree.
• Implementing a policy succession may be the most difficult portion of policy process. This may be true for several reasons:– Field staffs may not respond to nominal changes in policy; this may
result from inertia or a lack of understanding.– Modifications in one policy may adversely affect other policies and
organizations.– Changes may occur that go beyond the intended policy modifications.
Unit 9 Evaluation and Policy Change
27
Policy Succession• “New governance” via third-party and indirect mechanisms means
that connecting programs with outputs is more difficult; this may further inhibit policy succession.
• Policy succession takes massive political efforts, yet cannot show results for some time—it may therefore create disappointment in the new program.
• Policy successions may generate multiple effects on entrenched interests, resulting in more change than was intended.
• Because policy succession is difficult, it is appropriate to think about how to facilitate ongoing policy change when designing organizations and programs.
• Policies are increasingly being designed with built-in “terminators”:– Sunset laws and the declining sense of entitlement may encourage
continual change.– It is not possible to reverse history and add terminators to existing
programs that are already functioning without them.
Course Conclusion
• Unit 10 completes the course
• Follow-up in Discussion
• Make sure your Final Project is submitted
• I hope you enjoyed the course and will find it useful in future studies and work
• Good Luck and Good Night
Unit 9 Evaluation and Policy Change
29