Evaluation in Complex Policy Systems - ULiege Evaluation in Complex Policy Systems 435 05 Sanderson...

Evaluation in Complex Policy Systems

I A N S A N D E R S O NPolicy Research Institute, Leeds Metropolitan University, UK

The renewed emphasis in the UK on ‘evidence-based policy making’ hassharpened the focus on the utilization debate in the evaluation community.Traditionally, the emphasis is placed on methodological concerns but thisarticle argues for a sharper focus on the underpinning theoretical bases ofpolicy evaluation, both in terms of its role in policy making and in terms ofthe substantive theories which inform policy development andimplementation. In particular, the article seeks to assess the implications ofcomplexity theory for our theoretical assumptions about policy systems. It isargued that, together with ‘new institutionalist’ thought and recent work onpolicy implementation structures, notions of complexity have substantialramifications for the way in which we approach policy evaluation, given thecontemporary concern to address ‘cross-cutting’ social problems through‘joined-up’ policy initiatives. It is suggested that our thinking about evaluationreflects the broader reaction against ‘modernist’ conceptions of the role ofsocial science in our quest to change and improve the world.

KEYWORDS : complexity theory; cross-cutting issues; epistemology; newinstitutionalism; organizational theory

Introduction

The present Labour government has demonstrated a real determination to mod-ernize public services, evident in an agenda of change for local government,schools, the health service and, indeed, the whole machinery of central govern-ment. Key elements of the modernization agenda are: first, ensuring that policymaking is more forward looking, joined-up and strategic; second, making publicservices more responsive to the needs of users; and third, delivering services thatare of high quality and efficient (Cabinet Office, 1999). The process of change is,on the face of it, more collaborative and less ideologically-driven than in the past,more informed by pilot projects to test out new approaches and based upon a newpragmatism captured in the ministerial mantra ‘what matters is what works’(Martin and Sanderson, 1999).

The government has emphasized repeatedly the importance of evidence-baseddecision making and this was underlined by the Secretary of State for Educationand Employment in a recent speech to the Economic and Social Research

EvaluationCopyright © 2000

SAGE Publications (London, Thousand Oaks and New Delhi)

[1356–3890 (200010)6:4; 433–454; 015636]Vol 6(4): 433–454

433

05 Sanderson (to/d) 22/12/00 9:35 am Page 433

http:\\www.sagepub.co.uk

http:\\www.sagepub.co.uk

Council (ESRC) in the UK. ‘Rational thought is impossible without good evi-dence’, he argued; research is needed ‘to tell us what works and why, and whattypes of policy initiatives are likely to be most effective . . .’ and ‘. . . must vastlyimprove the quality and sensitivity of the complex and often constraineddecisions we, as politicians, have to make’ (DfEE, 2000: 4, 24, 25). The newCentre for Management and Policy Studies in the Cabinet Office is intended toprovide a ‘window at the heart of government’ for research and evaluation evi-dence (Cabinet Office, 2000). There are indeed examples of policies beinginformed more by evaluation findings (see Hasluck, 2000 on the EmploymentNew Deals) but there continue to be counter-examples (see Hutton, 1999 onpenal policy for young people). Moreover, there is always the danger that govern-ments will use the notion of ‘evidence-based decision making’ primarily tolegitimize their policies, using research evidence only when it supports politically-driven priorities (Kogan, 1999).

More generally, there is a long history of commentary on the failure of socialscience research, and evaluation research in particular, to live up to its promiseof contributing to the development of more effective public policies to addressour economic and social problems and improve our quality of life. Writers suchas Carol Weiss (1995), Thomas Cook (1985) and Michael Quinn Patton (1986)have addressed this issue, the latter referring to the ‘utilization crisis’ in evalu-ation. According to Pawson and Tilley (1997: 2–3), ‘evaluation research has notexactly lived up to its promise’ engendering ‘a sense of impotence when it comesto influencing real decision making’.

An important theme in the response to this problem focuses on the methodo-logical approach of evaluation, criticizing especially its positivist foundations andexperimentalism. Although considerable emphasis has been placed on the needfor theory-driven evaluation, the primary focus has been on epistemological con-cerns and the methodological strategies required to generate the understandingof how policies and programmes work, which is needed to inform better decisionmaking. Others have emphasized the importance of the ‘substantive’ theory thatunderpins the evaluation of public policies and programmes, highlighting thelimitations of rational choice models and the need to address implementationprocesses, implying, in effect, the incorporation of organizational theory. In par-ticular, a renewed focus on the role of institutions in mediating policy implemen-tation has emphasized the complexity of policy systems and this resonates withrecent work on complexity theory.

This article focuses on the theoretical bases of policy evaluation and in par-ticular on the implications of ‘complexity theory’ for the capacity of evaluationto promote policy learning. It comprises three main parts. The first considers therelationship between evaluation and policy making, in particular the dominantepistemological and theoretical assumptions and how these constrain the poten-tial of policy evaluation to inform learning and improvement in complex policysystems. The second part considers recent work on complexity theory and thechallenge that this poses to the assumptions upon which much analysis of theformulation and implementation of policy and evaluation of its effects proceeds.In particular, I discuss how complexity relates to ‘critical realism’ and to the ‘new

Evaluation 6(4)

434


institutionalism’ and the notion of ‘implementation structures’ and the impli-cations for policy theories. The final part seeks to assess the implications forevaluation in the context of complex policy systems and a brief conclusion raisesthe issue of the institutional conditions for effective evaluation.

The Dominant Logic and Assumptions of Policy Evaluation

There has been extensive soul-searching amongst the evaluation communityabout the perceived lack of impact of evaluation research on the world of poli-tics and practice. Thus, Hellstern (1986: 279) argues that the optimism of the1960s and 1970s about the potential for rational action guided by scientific know-ledge has been shaken:

Science has lost part of its credibility . . . and the experimental mode of thinking hasgiven way to a more practical thinking stressing individual, idiosyncratic knowledge andcommon-sense experience, recognizing that all knowledge is based on values and restson critical assumptions regarding human information processing capabilities.

Meanwhile, somewhat paradoxically, the demand for and use of evaluationhave continued to grow despite a lack of evidence of its usefulness in improvingthe effectiveness of public policies and programmes (Albaek, 1995). This lack ofevidence has led to a shift of focus in the analysis of usefulness and a search fora rationale in promoting broader ‘enlightenment’ rather than instrumental use.Thus, it is now widely argued that evaluation research, and social science researchgenerally, is more often used in a conceptual rather than an instrumental way,reaching decision makers in unsystematic and diffuse forms, ‘percolating’ into thepolicy arena and influencing thinking about policy issues, providing policy makerswith ‘a background of information and ideas that affected the ways that theyviewed issues and the alternatives that they considered’ (Weiss, 1995: 141). There-fore, it is argued that to conceive of the relationship between evaluation andpolicy making in instrumental terms understates its influence in enhancing publicpolicy effectiveness.

Of course, there is a danger that this argument might be used to let the evalu-ation community off the hook – practitioners are actually ‘imbibing’ knowledgein subconscious ways even though they do not fully appreciate it. Needless to say,the practitioner community is unlikely to be fully convinced and may be justifiedin seeking ways in which the more direct impact of evaluation on public policiesand programmes can be enhanced. We therefore need to look at approaches thathave been adopted for policy evaluation to assess why they have failed to produceforms of knowledge which can inform action to improve policy effectiveness.Indeed, I would argue that most approaches are founded upon a set of dominantassumptions, which can be considered at three levels. First, at the epistemologi-cal level lie assumptions about the validity of different forms of knowledge;behind different approaches to evaluation lie normative stances on this issue –about whether we can derive ‘reliable’ knowledge about the social world, abouthow we can do this and about what form such knowledge takes. The second levelcomprises assumptions about how policies are formulated and implemented –

Sanderson: Evaluation in Complex Policy Systems

435


about who are the ‘agents’ who make and implement policy and how they act,and about the factors which influence such action. At the third level, assumptionsare embodied in ‘substantive’ theories behind policies that address specific econ-omic and social problems. Such assumptions relate essentially to the way in whichsuch problems are defined and specified, the relationship between problems andpolicy instruments or measures, and the way in which the targeted agents respondto such instruments or measures.

The Epistemological Basis of EvaluationAs regards the epistemological basis of evaluation, much has been written aboutthe dominance of positivist assumptions such that the ‘general logic’ of evaluation(Schwandt, 1997) privileges the notion of objective, value-free knowledgederived through quantitative social science methodologies (Albaek, 1995;Drewett, 1997; Guba and Lincoln, 1989; Laughlin and Broadbent, 1996; Vander-plaat, 1995; Van der Knaap, 1995). Albaek (1995) argues that the dominance ofsuch assumptions has been broken down as the influence of positivism in the phil-osophy of science has declined and as the phenomenological and hermeneutic tra-ditions have gained support, providing growing credibility for qualitativeapproaches. However, Albaek may underestimate the hold over the evaluationfield that is exercised by what Schwandt (1997) calls the ‘modernist paradigm ofreason’. Indeed, recently Shaw (1999) has argued that the new enthusiasm for evi-dence-based approaches to public policy (which is particularly marked in thepresent government’s often-stated concern with ‘what works’ – see Martin andSanderson, 1999) has given a new lease of life and purposefulness to ‘preoccu-pations with measurement, traditional worries regarding reliability and validity,and other concerns captured within quantitative methodology’ (Shaw, 1999: 3).

Within such a context, qualitative approaches continually struggle to be seenas anything other than second best, to be used when there are problems opera-tionalizing quantitative designs or in a supplementary way to investigateimplementation processes (Shaw, 1999). Not only are they seen as failing toprovide hard, reliable, factual data, their status is also judged relative to the per-ceived ‘gold standard’ experimental approach to establishing causal attribution.Thus, in ideal circumstances an experimental (or quasi-experimental) design willbe able to isolate the net additional effect attributable to a policy or programme,allowing for deadweight (Rossi and Freeman, 1993). If it is not feasible to estab-lish control or comparison situations, the before-and-after design is seen as thefirst fall back, providing a quantitative estimate of gross effect and requiringsophisticated statistical techniques to estimate deadweight. Again, in this picturequalitative approaches are seen as providing supplementary analysis ofimplementation processes, in cases where an understanding of such processes isneeded, and the treatment of them as a ‘black box’ by (quasi-)experimentaldesigns is seen as problematical (Pawson and Tilley, 1997; Schmid, 1997).

Many commentators have argued that the lack of influence of evaluation onpolicy making can be attributed to the failure of this ‘dominant logic’ to producerelevant and useful information and that qualitative methods hold the promise ofmore effective utilization (e.g. Albaek, 1995; Hellstern, 1986). However, as

Evaluation 6(4)

436


Hellstern (1986: 304) acknowledges, problems arise since qualitative methods‘lack preciseness in disentangling effects and confound situational with systemiceffects; so there are no clear results to be used for programme design’. Pawsonand Tilley (1997) also emphasize this problem of the ‘context-boundedness’ ofconstructivist evaluation findings, which results in an ‘inability to grasp thosestructural and institutional features of society which are in some respects inde-pendent of the individual’s reasoning and desires’ (p. 23). Consequently, whileadding to our understanding of implementation processes and the role of differ-ent interests (or stakeholders), constuctivist approaches (like experimentalism)ultimately fail to provide generalizable findings which can guide policy makers asto what is likely to work (or not) in particular practical situations.

This problem, of course, is the familiar one of external validity – the extent towhich findings from a particular evaluation study can be generalized to other situ-ations. According to Shaw (1999: 86), this requires ‘ . . . the understanding andexplanation of mechanisms operating in a local context, in order that plausibleinferences can be drawn regarding other settings, people and interventions thatare of interest to policy makers’. As Pawson and Tilley (1997: 26) argue, the basisfor understanding ‘what it is about a programme which makes it work’ is a theory-based approach to evaluation. Theories comprise propositions about how poli-cies and programmes generate or produce effects and outcomes and an importantpurpose of evaluation (although as Julnes et al. [1998] point out, not the onlypurpose) is, in effect, to test these propositions to improve the evidence-base fordrawing the ‘plausible inferences’ about what is likely to work elsewhere.

However, much policy evaluation proceeds without explicit reference to thetheoretical propositions and assumptions upon which the development andimplementation of policies are based. This results in an implicit adoption of pre-vailing, taken-for-granted theoretical frameworks which, I would argue, furtherundermine the capacity of evaluation to produce knowledge which is useful ininforming social action. This can be considered in relation to the two levels oftheory identified above, viz. on the one hand theories about how policies aredeveloped and implemented and on the other hand theories about how specificpolicies and programmes generate outcomes and effects.

Theories of Policy Making and OrganizationThe field of policy evaluation is characterized by a wide variety of approaches,due to ‘infinite potential configurations of purpose, technique, concepts, actorsserved and institutional forms’ (Hellstern, 1986: 290). However, the taken-for-granted, common sense approach to policy evaluation derives its logic from theconceptualization of the process of policy formulation and implementation as arational cycle of goal specification, design, implementation, evaluation and re-design – what Jenkins-Smith and Sabatier (1993) call the ‘stages heuristic’. AsColebatch (1998: 42) argues: ‘The dominant paradigm in the study of policy seesit as the exercise of authority to achieve collective purposes. Policy is to be under-stood in terms of the pursuit of goals: our policy is what we want to achieve’. Itappears to be rational common sense to see policy as a purposive course of actionin pursuit of objectives based upon careful assessment of alternative ways of


437


achieving such objectives and effective implementation of the selected course ofaction. Moreover, rationality is enhanced by being clear about the objectives wewish to achieve, and by evaluating the extent to which the policy as implementedactually achieves these objectives. If policy is goal-driven, evaluation should begoal-oriented. Such evaluation completes the cycle and provides feedback toimprove the policy (Colebatch, 1998: 42–3; Parsons, 1995).

Within this rational model, then, evaluation fulfils an essentially ‘instrumental’function in answering the question: how effective are the chosen means in achiev-ing the specified ends? Schwandt (1997) situates this general logic firmly in themodernist paradigm of reason within which rationality is a matter of correct pro-cedure or method in a context where ‘policymakers seek to manage economicand social affairs “rationally” in an apolitical, scientized manner such that socialpolicy is more or less an exercise in social technology’ (Schwandt, 1997: 74). Anumber of key characteristics of evaluation can be seen as deriving from thisparadigm.

Most policy evaluations are commissioned by government departments andagencies and publicly-funded bodies accountable to them for the effective develop-ment and implementation of the policy. This has given a strong top-down orien-tation to evaluation with a major concern to promote accountability and control.As Henkel (1991: 20) argues, evaluation has been promoted by government:

. . . as a contribution to the control of the periphery by the centre, particularly in themanagement of resources. It stressed values of economy, efficiency, ‘value for money’and effectiveness of performance. As a corollary, it assumed that evaluation would besummative, delivering authoritative judgements, based as far as possible on perform-ance indicators or quantitative measures of input-output relationships and outcomesand set against predefined targets and standards.

Kettunen (1994) argues that top-down evaluation is concerned to strengthenthe prevailing politico-administrative and managerial structures. Its quantitativefocus can be seen as circumscribing the normative basis of evaluation, restrictingattention to the values and interests embodied in the formal official goals andobjectives of policies and programmes. As Hill (1993: 3) argues, the dominanceof such values is promoted because ‘much of the research has been commissionedby clients who expect prescription . . .’ and ‘. . . the top can often lay claim to legit-imacy for its goals, deriving from democratic policy making processes’.

Gray and Jenkins (1995: 128) have emphasized that evaluation is a product ofpolitical circumstances and the approach is contingent upon ‘types of politicalsystem and policy style’. They argue that its function can be ‘to close off issuesfrom public debate or to narrow the focus of discussion to questions that are seenas legitimate within a wider set of policy assumptions that themselves remainunquestioned and unexamined’. The political context in which the use of evalu-ation has developed in the UK is one in which a key motivation is the promotionof compliance in implementation with central government policy objectives.Policy evaluation therefore tends to be undertaken within a broader model ofpolicy reform which is highly prescriptive in terms of measures to improve top-down implementation; in Sabatier’s (1997) terms, what we have is ‘not entirely

Evaluation 6(4)

438


disinterested evaluation’. In this context, the role of evaluation as a normativeexercise in ‘values inquiry’ is restricted; rather, its ‘technical–instrumental’ role isreinforced, the views of other stakeholders being confined to matters of effectiveimplementation.

A further key characteristic of the dominant rationalist-modernist paradigm isthe tendency to emphasize methodology rather than theory as the basis of ‘good’evaluation. Thus, the argument goes, we need clearly defined objectives andintended outcomes, reliable ways of measuring them and sound methodologicaldesigns for identifying the effects attributable to policy interventions. As dis-cussed above, the experimental design is seen as the gold standard of evaluationand ‘it is not necessary to understand how a social programme works in order toestimate its net effects through randomized experiments’ (Chen and Rossi,quoted in Pawson and Tilley, 1997: 26). Schmid et al. (1996: 2) argue that in con-ventional top-down approaches to policy evaluation in the labour market fieldthere has been a strong focus on individual policy instruments and programmesand ‘ . . . policy formation and implementation processes have been treatedlargely as a black box’. This has resulted in a neglect of the task of explanation –of seeking to understand links and interactions between policy interventions, thecumulative impact of policies and the influence of institutional regimes. More-over, Schmid (1996, 1997) further suggests that although there has been a recog-nition of the need to ‘open up the black box’ and investigate the delivery ofpolicies and programmes, the approach to process evaluation has been inade-quate in addressing the effects of political, institutional and organizational con-texts on processes of policy formulation and implementation.

Part of the problem, then, is neglect of such political, institutional and organiz-ational processes; but, even to the extent that the influence of such processes isacknowledged, problems arise due to the nature of dominant theoretical frame-works for the study of political action and organizations, which frame the assump-tions that we make about the significance and operation of policy implementationprocesses. Taking theories of politics first, an influential critique of the con-ventional wisdom has been developed from the perspective of the ‘new institu-tionalism’ most notably associated with March and Olsen (1984, 1989). Theyargue that modern political science abstracts the explanation of political actionfrom its institutional context. Political institutions are not seen as causal factorsin the ordering of collective life. Political phenomena tend to be reduced to indi-vidual behaviour abstracted from organizational structures and rules. Moreover,such individual action is seen as the product of calculated decision making moti-vated by self interest rather than by rules, expectations, obligations, duties andtraditions embedded in organizations. Finally, such decision making is seen asinstrumentally concerned with allocating resources and achieving outcomes,rather than with promoting certain purposes and values associated with a ‘better’society.

Although there has been increasing recognition of the importance of insti-tutional and organizational contexts in the study of the formulation andimplementation of policy, nevertheless we again meet a set of ‘domain’ assump-tions in mainstream organization theory founded upon notions of rational choice


439


and instrumental action. Thus, as Thompson and McHugh (1995: 10) argue, atechnical conception of organizations as rational, goal-seeking instruments pro-duces an ‘ . . . emphasis on rationally designed structures and practices resting onprocesses of calculated planning which will all maximise organizational effec-tiveness’. Moreover, they argue that mainstream theory is strongly influenced byideas of organizations as co-operative social systems in which all elements play afunctional role, with self-regulating processes tending towards a state of stableequilibrium and order (Thompson and McHugh, 1995: 11). Such a technical con-ception tends to abstract organizations from their institutional and social context;in relation to the problem of understanding policy implementation it focusesattention on the role of management action based upon rational analysis and stra-tegic choice (cf. Alvesson and Willmott, 1996). Effective implementationbecomes synonymous with effective management and notions of effectivemanagement are abstracted from the social relations of organizations. The influ-ence of power and routine is neglected; as Thompson and McHugh (1995: 14)state, ‘ . . . many deep-rooted features of organizational life – inequality, conflict,domination and subordination, manipulation – are written out of the script . . .’.

Policy and Programme TheoryThe third level of theory that is important in the consideration of approaches topolicy evaluation concerns notions of how policies and programmes actuallyproduce (or influence the production of ) effects and changes amongst the targetagents. Leeuw (1995) uses the term ‘policy theory’; Pawson and Tilley (1997)prefer ‘programme theory’. According to Leeuw (1995: 20):

A policy theory is a system of social and behavioural assumptions that underlie a publicpolicy which have been reformulated in the form of propositions. These propositionsreflect the beliefs of policy makers about the cognitions, attitudes, and behaviours ofthe policy’s target group: the people whom the policy is to affect.

To take an example from the field of labour market policy, over the years poli-cies to address unemployment have been based upon propositions concerning therelationship between lack of skills and unemployment and the effect of trainingon the skills and attitudes of the unemployed and therefore on their employabil-ity. In theory-driven evaluation, an important task is to seek to test such propo-sitions in order to improve our understanding of how policies and programmeswork, and thereby inform the development of more effective policies. However,we have indicated that much goal-oriented evaluation is concerned more withestablishing whether policies and programmes achieve their intended effectsrather than with understanding how they achieve their effects (unintended as wellas intended), and this has limited its contribution to policy learning.

Moreover, conventional approaches to policy evaluation tend to focus onspecific individual policy instruments and programmes where the policy theory canbe specified in a ‘controllable’ way (cf. Schmid, 1997). Where such instruments andprogrammes are implemented in similar organizational contexts, such an approachallows the influence of such contexts to be controlled out (Hjern and Hull, 1982:107). In addition, this promotes a focus on the effects of policy instruments and

Evaluation 6(4)

440


programmes at the level of individual agents (i.e. individual persons, householdunits, firms etc.). Such a focus is a requirement of experimental approaches inwhich the unit of analysis must be reasonably homogeneous in terms of the factorsinfluencing response to policy instruments.

Indeed, much of the theoretical underpinnings of public policy are based uponthe notion of influencing the behaviour of individual agents. Many of the basicpremises derive from neo-classical economic theory, which is premised upon theassumption of rational, utility maximizing economic agents (individuals andfirms). As Ormerod (1998) argues, the theory can provide reasonable approxi-mations in limited contexts, when the influence of broader factors influencingpreferences can be controlled out. However, this limits its general applicability inexplaining real-world economic processes. We can conceive of the abstractedtests of competitive equilibrium theorems as having equivalence with experi-mental evaluations of the effects of specific programmes on targeted individuals,again abstracted from a range of factors and thus lacking in external validity.

Therefore, we can see that the dominant theoretical underpinnings of policyevaluation embody some common basic assumptions about the way in whichpolicy is formulated and implemented and about the way in which policies effectchange in social systems. Two key assumptions can be highlighted. First, theassumptions of rational choice underpin the notion that we know how to changesocial systems to achieve desired ends; such ends are seen as consensually agreedand capable of clear definition such that we know what we want to achieve. Objec-tive social scientific knowledge of causes and effects provides the basis for confi-dence that ‘correct’ policies can be specified ex ante. Second, the assumptions ofNewtonian ‘social physics’ (Marion, 1999) underpin the notion that we have thecapacity to change social systems. The key assumptions relate to linear, succes-sionist causality, stability and symmetry in relationships between variables and aresulting proportionality of change in response to policy intervention. Theseassumptions provide confidence in a capacity to control social systems and anability to predict the consequences of policy interventions.

Based upon these assumptions, it is possible to see how policy evaluationbecomes oriented to assessing ex post facto the extent to which policy interven-tions have achieved their intended effects. The role of evaluation is to providefeedback in a basically stable and harmonious systems framework to guide ourquest to change social systems for the better and achieve agreed ends. Theproblem is that it has not been very successful in this role and this can be attrib-uted in large part to problems with the basic underlying assumptions. A majorchallenge to these assumptions is posed by recent work on complexity theory. Thenature of this challenge and the implications for our understanding of complexpolicy systems are considered in the next section.

Complexity, Realism and Institutions

The concepts of chaos and complexity are now well established in the natural sci-ences but have only recently been incorporated in the social sciences. Recentpublications on which this account draws are Byrne (1998); Elliot and Kiel (1996);


441


Harvey and Reed (1996); Marion (1999); Ormerod (1998); and Reed and Harvey(1992). Notions of complexity have their roots in chaos theory:

Chaos theory is the result of natural scientists’ discoveries in the field of nonlineardynamics. Nonlinear dynamics is the study of the temporal evolution of nonlinearsystems. Nonlinear systems reveal dynamical behavior such that the relationshipsbetween variables are unstable. Furthermore, changes in these relationships are subjectto positive feedback in which changes are amplified, breaking up existing structures andbehavior and creating unexpected outcomes in the generation of new structures andbehavior. These changes may result in new forms of equilibrium; novel forms of increas-ing complexity; or even temporal behavior that appears random and devoid of order,the state of ‘chaos’ in which uncertainty dominates and predictability breaks down.

(Elliot and Kiel, 1996: 1)

Marion (1999: 6–7) argues that while chaos theory is more suited to naturalsystems, notions of complexity can take account of ‘ . . . such issues as adaptation,deliberative behavior, intelligent behavior, reproduction and evolution . . .’ insocial systems and allow some degree of stability and inertia. However, as Reedand Harvey (1992: 359) state, complex systems ‘ . . . defy standard positivistcanons of description, prediction and explanation . . .’; their behaviour resultsfrom dynamic interaction between their component parts over time and emergesas the ‘holistic sum’ of these interactions. Functionalist notions of order-seeking,near-to-equilibrium social arrangements are inappropriate; rather, it is argued,the principles of ‘dissipative systems’ are applicable to the analysis of sociallife.

Dissipative systems are the subject of study in non-linear thermodynamics.Because such systems are thermodynamically open, they are capable of assimi-lating large amounts of energy from their environments and converting this intoincreasing structural complexity. This process is irreversible so dissipative systemspossess an evolutionary capacity (based on so-called negentropic processes)which enables them to avoid reversion to a state of thermodynamic equilibrium.Dissipative systems which can sustain these processes supporting the evolutionof increasing structural complexity can, in certain circumstances, undergo sudden,rapid, transformational change (a so-called bifurcation) and evolve to a new state.The achievement of such transformational change is very sensitive to the appro-priate conjunction of ‘initial conditions’ and is therefore non-replicable over time;knowledge of a system’s behaviour in the past will provide little guide to likelyfuture behaviour (Elliot and Kiel, 1996: 5–6; Harvey and Reed, 1996: 320–5; Reedand Harvey, 1992: 361–5).

The problems with prediction, however, do not imply that we cannot explainthe behaviour of dissipative systems; according to Reed and Harvey (1992: 364)‘ . . . we can at least reconstruct the particular constellation of structured choiceand accident that led to the present reality’. But approaches founded upon theassumptions of stability and equilibrium, of linearity in the relationship betweenvariables, and of proportionality of change in response to causal influences – suchapproaches are not appropriate in seeking to understand social systems thatexhibit complexity. The implications of complexity are highly problematical for

Evaluation 6(4)

442


positivist social science because empirical regularities based upon observablesocial phenomena are likely to be highly misleading representations of the wayin which complex systems actually work. Harvey and Reed (1996) argue that criti-cal realism provides an appropriate ‘philosophical ontology’ for the study of com-plexity and this is endorsed enthusiastically by Byrne (1998).

The tenets of critical realism now have widespread currency in the evaluationcommunity due to the work of Pawson and Tilley (1997). Very briefly, it is arguedfirst that ‘ . . . science is anchored in an intransitive domain, in a world whoseautonomous constitution stands independent of the knowing subject . . .’ (Harveyand Reed, 1996: 298). Second, this intransitive world comprises nested layers ofreality and the purpose of scientific endeavour is to penetrate to ever deeperlevels to uncover the causal factors behind phenomena which can be analysed atany one level. Third, these layers of reality comprise ‘ . . . a generative nexus ofmechanisms or entities . . .’ which are not immediately observable but ‘ . . . areendowed with real causal powers, latent capacities and slumbering liabilities’(Reed and Harvey, 1992: 356). Such entities or mechanisms generate socialphenomena ‘ . . . but not in the clean, recurrent stream of cause and effect that isregularly manufactured in the laboratory’ (Reed and Harvey, 1992).

In the context of social systems, Reed and Harvey (1992) argue that criticalrealism can be aligned with the notion of dissipative social systems by adoptingBhaskar’s (1998) ‘social naturalism’, which presents a ‘transformational model ofsocial action’. This model posits three levels of social reality: first, ‘society’ as a‘structural entity’; second, the individual, subject to socialization in particularsocial contexts but nevertheless with significant powers of agency; and third, anintermediary level comprising a ‘position-practice system’ of ‘rules, roles andrelations’ that regulate the interactions between the individual and social levels.Thus, individuals’ interpretations, intentions and actions are conditioned bysocial structures and relations and by institutionalized rules and motivations; yetit is through human agents that society is reproduced and changed (see alsoArcher, 1995:135–61). Indeed, it is this power of agency to generate ‘emergent’change of an indeterminate nature in social systems which underpins the notionof dissipative social systems in which there is ‘ . . . not so much convergence andconformity, but a diversity of unpredictable outcomes . . .’. Reed and Harvey(1992: 371) conclude:

Thus, optimal adaptation, equilibrium, and the convergence of evolutionary forms, solong the conceptual staple of Parsonian theory, need no longer be considered thenatural outcome of social action, nor an inevitable (if unanticipated) outcome of socialsystem functioning. Instead, the social ontology described here is grounded in a worldview that presumes the naturalness of both disjunctive development and of asym-metrical structures that carry within themselves inherent tendencies for periodic self-negation and radical change.

An important characteristic of the above framework is that it transcends the tra-ditional dualistic treatment of society/individual, structure/agency, macro/microetc., by emphasizing the role of an intermediary level that regulates perception,interpretation and action at the individual level. It is this level that is the focus of


443


‘institutionalist’ approaches; as Lowndes (1996: 182) argues: ‘Institution is amiddle-level (or “meso”) concept’. Institutions can be defined as:

. . . rules used by individuals for determining who and what are included in decisionsituations, how information is structured, what actions can be taken and in whatsequence, and how individual actions will be aggregated into collective decisions . . . allof which exist in a language shared by some community of individuals rather than asthe physical parts of some external environment. (Kiser and Ostrom quoted byDowding, 1994: 108)

The ‘new institutionalism’ embodies a critique of the individualistic rationalchoice focus of social and political theory and the concern with the influence ofreason (rationality and intentional action) and competition/coercion (power andbargaining). Thus, March and Olsen argue for ‘ . . . a more autonomous role forpolitical institutions . . .’ (1984: 738) and emphasize the need to investigate theinfluence of internal institutional processes, of the ‘normative order’ of duties,obligations, roles and rules, of career structures and professional standards, andof the ‘symbolic order’ of myths, symbols, rituals and ceremonies (March andOlsen, 1984, 1989). The new institutionalism, then, provides a focus on a ‘level’of political and social life which has been neglected in modern theory. As Fried-land and Alford (1991) and Lowndes (1996) argue, it focuses on a ‘middle level’of analysis between macro-social structures and processes on the one hand andthe micro-level of social agents (individuals, households, firms etc.) on the other.Institutions comprise both material and symbolic elements – both formal ‘rulesof the game’ and informal values, norms, interests, incentives and taken-for-granted beliefs. These shape social action, imposing constraints while also pro-viding opportunities, but may be the subject of conflict and struggle betweencompeting interests (DiMaggio and Powell, 1991).

Friedland and Alford’s (1991) notion of society as an ‘interinstitutional system’resonates with Bhaskar’s framework which emphasizes the complexity of multi-level social reality. The focus on the role of organizations and institutions isreflected in recent work in the study of policy making which emphasizes the com-plexity of implementation processes in contrast to the traditional ‘top-down’focus on the specification and achievement of goals, on reasons for ‘implemen-tation failure’ when goals are not (fully) achieved, and on how compliance withpolicy makers’ intentions can be improved. The so-called bottom-up view focuseson the network of actors and agencies involved in implementation – what Hjernand Porter (1997) call the ‘implementation structure’ – and on processes ofnegotiation and bargaining which affect outcomes. It is this institutional andorganizational context ‘ . . . where certain, non-constitutional rules of behaviourprevail . . .’ (Kettunen, 1994: 36) that is essentially treated as a black box in thetop-down approach (Hjern and Hull, 1982).

Evaluation and Complex ‘Cross-Cutting’ Policy Issues

If we accept this conception of complex, multi-level, dynamic policy systems, thenthere are significant implications for policy evaluation. First, we need to re-assess

Evaluation 6(4)

444


our traditional modernist assumptions about our capacity to control and guidesocial systems through public policy interventions. If we are to accommodate theimplications of complexity in models of policy making and management that arecapable of guiding action to achieve collectively agreed desired social outcomes,then we need to find some terra firma between, on the one hand, the illusion ofcertainty in modernist-rationalist order and, on the other, the danger of a pes-simistic nihilism when facing chaos and complexity. Ormerod (1998) argues thatthe focus of collective action needs to shift:

Despite the difficulties involved in managing a complex world, governments still havean important role to play. They should do very much less in terms of detailed, short-term intervention. And they should spend much more time thinking about the overallframework of whatever particular problem is at issue. For it is here that governmentshave the potential to achieve a great deal. Less can be more. (pp. 183–4)

For situations in which we lack the basis for confidence that policy decisionswill be correct ex ante, or that such decisions will be implemented as intended,Dunsire (1986) argued some years ago that the answer may lie, first, in strength-ening the role of evaluation to provide up-to-date, relevant information on actualperformance and, second, in building capacity to take action to modify policydesign and implementation on the basis of such information. Kaufmann (1986:224) supports this view: ‘ . . . it is more important to make learning processes poss-ible than to make the best decision in advance’. What this amounts to is buildingthe capacity for policy learning and adaptation based upon evaluation. Indeed,we have noted the enhanced commitment of the present government to piloting,prototyping and evaluating policy initiatives to address what are seen as key econ-omic and social problems requiring joined-up solutions, but evaluation is focusedmore on ‘identifying what’ than on ‘explaining how’.

In order to provide the capacity for learning in complex policy systems, evalu-ation must become a more ‘exploratory’ and ‘explanatory’ enterprise, to captureand understand system change in response to policy interventions, which is non-proportional and non-linear. Thus, little change may be evident in a system for along period but then major change may suddenly (and unpredictably) occur as‘synergy’ occurs between a number of pre-conditions. Our prior causal modelsmay not cover all important effects so we need to search for unintended or unan-ticipated effects. We also need to ensure that such models incorporate the medi-ating institutional and organizational levels in explanations of how policy effectsare brought about in practice. Approaches to evaluation which seek to isolatepolicy instruments or programmes in controlled situations will produce results oflimited usefulness because they are context-bound, lacking the basis for general-ization to guide action in other contexts.

These limitations are especially evident in the face of the evaluation challengepresented by the current policy emphasis on complex ‘cross-cutting’ issues, suchas crime, health, environmental sustainability, unemployment and social exclu-sion, and on promoting joined-up policy responses to such issues. Thus, a keyconcern in developing policy responses to such issues is with the notion of‘adding value’. The logic behind this notion is that maximum effectiveness in


445


addressing complex policy problems can be achieved by co-ordinating and inte-grating mainstream policies, focusing them on the problems concerned, and pro-viding supplementary and additional measures and resources only where thesecan enhance the positive impacts and benefits of existing policies and pro-grammes. The emphasis, in adding value, is on avoiding duplication, on adapta-bility and flexibility, on ‘catalytic’ support, on innovative approaches and oncapacity building – that is, helping to build the capacity of mainstream agenciesto adapt and change their purposes, objectives, policies, programmes and waysof working to address these complex issues effectively through ‘joined-up’government.

Consequently, inherent in this notion of adding value is the objective of achiev-ing a step change both in the economic and social systems which are the target ofpolicy interventions and in the capacity of government for effective intervention.We can see that the assumption here has affinity with the basic tenets of complexitytheory – that if the right circumstances (cf. initial conditions) can be created thenthe potential exists for promoting transformative change in economic and socialconditions. The right circumstances involve getting the relevant economic, socialand political agents to work together in a co-ordinated way within a strategicframework to create synergies and to provide catalytic resources to lever out thebenefits such that the whole becomes greater than the sum of the parts.

This policy context presents a number of implications for evaluation. First,faced with such complex policy situations, evaluators face substantial difficultiesoperationalizing theory-based approaches to evaluation. Such difficulties arise,first, due to conventional conceptions of the process of theory development as adeductive process of hypothesis testing; and second, from the practical constraintsfaced in specific evaluation contexts that often conspire against rigorous hypoth-esis testing. The usual response is to focus on the latter aspect of the problem andseek to encourage in evaluation practice, conditions which allow more effectiveimplementation of conventional evaluation designs, which are seen as providingthe sound basis for theory-based evaluation. However, the danger with thisresponse, of course, is that it results in a tendency to compartmentalize the evalu-ation of policies and programmes because of the need to ‘control’ the conditionsfor valid hypothesis testing. Therefore, in complex policy environments wherepotentially many programmes interact and hypotheses about their effects arelikely to be heavily context-dependent, the exhortation to methodological refine-ment may not get us very far.

Rather, as Schmid (1996: 205) has argued, we may need to give some attentionto our conceptions of theory development:

. . . the composition of the picture has to be improved by more explicit theory-buildingand theory-testing, although the current state of the art means the complexity of thesubject requires some deliberate eclecticism and modesty in terms of general theory;rather than aiming at an axiomatic system of nomological statements (i.e. statementsof universal laws), ‘pattern recognition’ and ‘pattern prediction’ is the realistic goal . . .

From this perspective, we need to recognize that our theory-building endeavoursin complex policy systems need to be grounded in a keen sense of humility and a

Evaluation 6(4)

446


recognition of the importance of ‘walking before we can run’. Therefore, descrip-tive approaches need to be given more weight in developing a better under-standing of how policies and programmes are implemented in practice; how theyactually generate effects and change over time, and how such effects depend oncontextual circumstances and inter-relationships with other policies and pro-cesses. Comprehensive ‘policy mapping’ can provide the basis for ‘pattern recog-nition’ which in turn can generate hypotheses for testing. For example, the use of‘concept mapping’ has been advocated by Knox (1995) in complex policy con-texts and this is endorsed in the European Commission’s work on evaluationthrough the MEANS programme (European Commission, 1999). However, theprocess must be self-reflective, recognizing that description is always dependentupon some prior theorizing. Moreover, it must be recognized that the process oftesting will often not conform to the purist tenets of the hypothetico-deductivemethod but rather will involve a much more eclectic process in relation to bothsources and types of evidence, and the forms of judgement employed about themeaning and implications of such evidence.

Considerations relating to theory development raise the issue of context depen-dency, which is of central importance in the theory of complex systems and in therealist social ontology which we have discussed. Thus, evaluation in the context ofcomplex policy systems must cope with the ‘embeddedness’ of such systems withina wider range of social processes (cf. Pawson and Tilley, 1997). Individual com-ponents of policy interventions cannot be evaluated effectively if they are ‘lifted’out of this context; it is necessary to identify and understand the influence of thekey contextual factors and also the ‘historical’ nature of the task. Thus, the way inwhich contextual factors impact upon a policy may depend crucially upon howthey interact at a particular juncture in time. Again, complexity theory suggeststhat the effect of a policy intervention may be due to the fact that it was ‘fortu-itously’ introduced at a particular point in time into a particular set of circum-stances such that it provided the catalyst for a step change in the system. A keyelement in evaluation to understand the influence of context will be comparativeanalyses over time of carefully selected instances of similar policy initiatives imple-mented in different contextual circumstances. Under such conditions, of course,controlled (quasi-)experimental comparisons are unlikely to be feasible.

Moreover, policy interventions are always ‘ . . . embedded in a larger “policylandscape” . . .’ (Schmid, 1996: 205) and this aspect of complexity is particularlyimportant in the current emphasis on ‘joined-up’ policy solutions. Approaches toevaluation must be able to accommodate complex, multi-programme initiativesin which there is limited potential to analyse specific measures in a controlled way.Thus, there is a need for holistic approaches which can identify how policies andprogrammes interact, causing tensions and conflicts or creating synergies; again,policy mapping is likely to be useful here. An issue of importance here relates tothe lesson from complexity theory that it is not possible to infer the (emergent)macro-level behaviour of a system straightforwardly from micro-level analysis. Inthe past, evaluation has focused too much on the effects of single programmesand on effects at the level of the individual agent and this has been a key factorin the problem of lack of external validity. In the context of complex policy


447


systems, if evaluation is to be useful in providing meaningful lessons about ‘whatmight work’ in various situations, it must be able to provide evidence of effectsat the macro-level.

However, the provision of such evidence is a long-term venture. Sabatier(1986) has identified the problem of ‘premature assessment’ of the effects of poli-cies and programmes, a problem which besets evaluation in many policy fields.He argues that a timeframe of 10–15 years is needed in order to identify how poli-cies and programmes are working as a basis for learning. In particular, such atimeframe is needed to be able to distinguish, on the one hand, the effects of theimplementation structure and, on the other hand, the validity of the relevantpolicy theory, the latter being crucially dependent in complex systems uponlonger-term impacts. Thus, there may be a long period of little discernible changein response to policy intervention before the conditions become right for synergyeffects to produce sudden rapid change. This creates a potentially seriousproblem for policy experiments or pilots in testing policy approaches to address-ing complex economic and social problems, as favoured by the present govern-ment (cf. Martin and Sanderson, 1999). In the relatively short timescale that ispolitically feasible for such pilots and their evaluation, it is likely that policyimpacts (especially at the macro-level) will not be discernible and that the focuswill necessarily be on implementation processes. Indeed, in several recent cases(e.g. the Employment New Deals and Best Value in local government) the UKgovernment has proceeded with policy commitments even though evaluation evi-dence from pilots has not demonstrated whether they work, but rather has mainlyprovided information to help make them work. Clearly, evaluation practice mustbe reconciled with ‘political realities’ but I would argue that greater priority needsto be given to the long-term project of enhancing the contribution of evaluationto the theoretical evidence base for improved policy responses to complex econ-omic and social problems.

A further issue deriving from the notion of holistic evaluation relates toassumptions about the methodological basis upon which it is possible to derivevalid evidence about the effects of policy interventions and the way in which theseare brought about. I have referred to the positivist versus interpretivist ‘paradigmwars’ and the realist position, particularly the social ontology based uponBhaskar’s work, which recognizes that different approaches and methods forsocial scientific enquiry have validity in analysing social processes operating atdifferent levels of social reality. Taking Harvey and Reed’s (1996: 297) position,‘ . . . the naturalism we advocate demands social scientists employ a methodo-logical pluralism when studying institutional life from a dissipative systems per-spective’. Broadly speaking, this position recognizes the value of quantitativemodelling and statistical and experimental approaches in recognizing andanalysing patterns and relationships between social phenomena, but sees this as,to some degree, ‘surface analysis’. Penetration to ‘deeper’ levels of analysisrequires an understanding of the way in which organizational and institutionalcontexts ‘structure’ decision making, action and behaviour by actors in a policysystem and in achieving this, qualitative methods must be seen as important.

Much of the scepticism about interpretivist approaches relates to the ‘strong

Evaluation 6(4)

448


version’ of constructivism, criticised by Pawson and Tilley (1997: 21) as ‘ . . . throw-ing out the objectivist baby with the relativist bathwater . . .’. However, it is clearthat hermeneutic approaches, which emphasize ethnographic interpretation andunderstanding, represent a vital element in the evaluator’s quest to understandprocesses operating at the institutional and individual levels of analysis. Thus, theyplay an important role in seeking to understand how formal laws, rules and pro-cedures and informal values, norms, conventions and beliefs operate in organiz-ations to structure behaviour and action, and how these are mediated throughstructures of power. Using such approaches, ‘ . . . by being witness to the day-to-day reasoning of . . . research subjects, by engaging in their life world, by partici-pating in their decision making, the researcher would be that much closer toreality’ (Pawson and Tilley, 1997).

There is now widespread support for the notion of methodological pluralismin evaluation as the grip of positivist thinking has been loosened and as it isacknowledged that there can be no methodological guarantees of truth, and thatwe need ‘ . . . to collect evidence from different angles and to construct an intel-ligible picture of the puzzle . . .’ (Schmid, 1996: 205). A further implication of thisposition is the need to include in the scope of evaluation all relevant stakehold-ers, recognizing their interest in the policy or programme and seeking to inter-pret their perspectives, theories of change, arguments and actions in relation totheir organizational, institutional and social context (Connell et al., 1995). Incomplex policy systems this implies the need for well-developed skills in trian-gulation on the part of evaluators and the ability to exercise carefully balancedjudgement in ‘constructing the intelligible picture’. Whatever the argumentsabout the role of judgement in evaluation, the fact is that evaluators of complexpolicy systems have to make decisions and judgements about the significance andweight of various sources and forms of evidence. It is important to recognize thisas one of the portfolio of competencies needed by evaluators, and to ensure thatit is based upon a sound, theoretically-grounded knowledge of the relevant policyenvironment and incorporates an adherence to credible ethical standards for theconduct of social science.

Of course, the case for stakeholder participation is not restricted to the cogni-tive dimension of evaluation; there is also a strong normative dimension to theargument, evident most clearly in advocacy of democratic evaluation (Floc’hlayand Plottu, 1998) and empowerment evaluation (Fetterman et al., 1996). Suchadvocacy is based, to a large degree at least, on a value stance which is critical ofthe way in which top-down, technocratic and instrumental approaches to evalu-ation can serve to reinforce the prevailing structure of power and alienate anddisempower further those disadvantaged groups whom public policies and pro-grammes are intended to help (Sanderson, 2000). The potential contradiction insuch approaches has been pointed up by Barr et al. (1996) in the context of pro-grammes to promote community development and capacity building, an increas-ingly important element in contemporary policy responses to social exclusion: ‘Itis inconsistent to invest in an activity which has the declared aim of respondingto community perceptions of need and appropriate action but impose criteria forevaluation of its performance which come from elsewhere’.


449


It is possible to distinguish two stances in arguments about this normativedimension. The first involves a more or less explicit value stance in favour of usingevaluation processes to empower participants, to encourage the development ofdemocratic values and to build capacity for participant self-evaluation (cf. Fetter-man et al., 1996); this is not addressed here. The second stance argues that evalu-ations must pose the question: ‘whose values are to count?’. In other words,evaluations should seek to identify and elaborate the values of all stakeholdersin a policy or programme and identify the implications and effects for each. InJulnes et al.’s (1998) view, one of the legitimate purposes of evaluation is ‘valuesinquiry’. This can inform future valuing of policies and programmes and impliesthat evaluation can contribute to learning in the normative as well as cognitivedimension. This stance has important implications in the context of complexsocial policy systems due to the potentially very wide range of stakeholdersinvolved (especially in cross-sectoral initiatives) and the considerable scope forvalue conflict in relation to the underlying issues.

Conclusion

We can see, then, that complexity theory has some important implications forevaluation in the context of policy initiatives to address key economic and socialproblems. It requires us to challenge some basic assumptions that underpin tra-ditional approaches to evaluation, concerning how we can gain access to andanalyse social processes, how policies to change such processes are formulatedand implemented, and how such policies ‘work’ in promoting change. It requiresus to recognize that evaluation is necessarily itself a highly complex endeavour ifwe accept the realist notion of a multi-layered social reality, the applicability ofthe concept of dissipative systems and the force of hermeneutic accounts of therole of human agency. It requires us to see evaluation essentially as a craft or‘practice’. This is not the application of ‘techniques’ to well-defined policy con-texts which will provide a definitive answer to the question ‘are our objectivesbeing achieved?’. Rather, it is more an exercise in crafting an approach compris-ing a range of methods appropriate to particular circumstances which will providesome understanding of the wider appropriateness of policy initiatives. Appropri-ateness can be seen as comprising three key elements: the first is identifying theeffects or consequences of a policy intervention in relation to different stake-holders; the second is understanding how those effects are brought about; and thethird is valuing the policy intervention, again in relation to the alternative nor-mative frameworks of the various stakeholders.

From this perspective, then, there is a need for some caution in the claims thatare made for evaluation as the basis for policy learning which can drive improve-ment in our capacity to change the world for the better. If we adopt the notionof evaluation as ‘practice’ rather than ‘technique’ then the issue of the insti-tutional framework for evaluation becomes fundamentally important. I haveargued that much evaluation takes place in the context of the rational, top-downperspective on policy making and management which, according to Degeling(1993: 48) ‘ . . . references and legitimates the authority and control over policy

Evaluation 6(4)

450


conventionally attributed to ministers and, by implication, their senior man-agers. . .’ and provides the basis for ‘ . . . asserting claims of hierarchy and for legit-imating the rational/instrumental prescriptions offered to bolster hierarchy as theremedies for gaps in implementation’. Such a context is clearly not conducive tothe vision of evaluation advocated here and it is difficult to change situations ofdomination, lack of trust and self-interested behaviour that are not conducive tolearning.

The appropriate reference, I would argue, is to the institutional conditions forthe development of social scientific knowledge more generally, in which thepotential for open communication and debate is fundamental. Van der Knaap(1995) emphasizes the importance of ‘communicative processes of argumenta-tion’ in policy learning, a position also adopted by Sabatier (1986: 323):

An important factor affecting the extent of policy-oriented learning seems to be theexistence of reasonably well-structured communication fora . . . in which professionalsand other experts from different advocacy coalitions are forced to confront each other’sarguments in a relatively depoliticized setting.

This position has clear affinities with Habermas’s (1984) conception of ‘idealpolitical discourse’ in which the rationality of consensus outcomes derives fromthe ‘force of better argument’ in conditions of ‘communicative rationality’ – opendiscussion and argumentation free from ‘distortions’ due to the coercive exerciseof power and ideology.

Although the prospects for overcoming the influence of existing power con-figurations and attaining the institutional procedures and conditions for ‘com-municatively rational discourse’ are, to say the least, debatable, there is alwaysthe potential to create ‘local fora’ in which broader power structures can be sus-pended. In such a context it may be that a common commitment to addressingbasic social problems and needs through effective collective action can create theconditions for learning. Ultimately, it may be our capacity to develop institutionalconditions of ‘communicative competence’ rather than our traditional preoccu-pation with ‘technical competence’ which is more important in our quest toharness evaluation more effectively to the task of addressing complex social prob-lems and improving social welfare.

NoteThe author wishes to thank Peter Van der Knaap, two anonymous referees and the journaleditor for valuable comments on an earlier draft of this article.

ReferencesAlbaek, E. (1995) ‘Policy Evaluation: Design and Utilization’, in R. C. Rist (ed.) Policy

Evaluation: Linking Theory to Practice. Aldershot: Edward Elgar.Alvesson, M. and H. Willmott (1996) Making Sense of Management: A Critical Introduc-

tion. London: Sage.Archer, M. S. (1995) Realist Social Theory: The Morphogenetic Approach. Cambridge:

Cambridge University Press.


451


Barr, A., S. Hashagen and R. Purcell (1996) Monitoring and Evaluation of CommunityDevelopment in Northern Ireland. Belfast: Department of Health and Social Services,Voluntary Activity Unit.

Bhaskar, R. (1998) ‘Societies’, in M. Archer, R. Bhaskar, A. Collier, T. Lawson and A.Norrie (eds) Critical Realism: Essential Readings. London: Routledge.

Byrne, D. (1998) Complexity Theory and the Social Sciences: An Introduction. London:Routledge.

Cabinet Office (1999) Modernizing Government, Cm. 4310. London: The StationeryOffice.

Cabinet Office (2000) Adding it Up: Improving Analysis and Modelling in Central Govern-ment. London: Central Office of Information.

Colebatch, H. K. (1998) Policy. Buckingham: Open University Press.Connell, J. P., A. C. Kubisch, L. B. Schorr and C. H. Weiss (eds) (1995) New Approaches

to Evaluating Community Initiatives. Volume 1: Concepts, Methods and Contexts.Washington, DC: Aspen Institute.

Cook, T. D. (1985) ‘Postpositivist Critical Multiplism’, in R. L. Shotland and M. M. Mark(eds) Social Science and Social Policy. Beverly Hills, CA: Sage.

Degeling, P. (1993) ‘Policy as the Accomplishment of an Implementation Structure: Hos-pital Restructuring in Australia’, in M. Hill (ed.) New Agendas in the Study of the PolicyProcess. London: Harvester Wheatsheaf.

DfEE (2000) Influence or Irrelevance: Can Social Science Improve Government? Speechby the Secretary of State to the ESRC. London: Department for Education and Employ-ment.

DiMaggio, P. J. and W. W. Powell (1991) ‘Introduction’, in W. W. Powell and P. J. DiMag-gio (eds) The New Institutionalism in Organizational Analysis. Chicago, IL: Universityof Chicago Press.

Dowding, K. (1994) ‘The Comparability of Behaviouralism, Rational Choice and “NewInstitutionalism” ’, Journal of Theoretical Politics 6(1):105–17.

Drewett, A. (1997) ‘Evaluation and Consultation: Learning the Lessons of User Involve-ment’, Evaluation 3(2): 189–204.

Dunsire, A. (1986) ‘A Cybernetic View of Guidance, Control and Evaluation in the PublicSector’, in F.-X. Kaufmann, G. Majone, V. Ostrom and W. Wirth (eds) Guidance,Control and Evaluation in the Public Sector. Berlin: Walter de Gruyter.

Elliot, E. and L. D. Kiel (1996) ‘Introduction’, in L. D. Kiel and E. Elliot (eds) ChaosTheory in the Social Sciences: Foundations and Applications. Ann Arbor, MI: Universityof Michigan Press.

European Commission (1999) Evaluating Socio-Economic Programmes: Principal Evalu-ation Techniques and Tools, MEANS Collection Volume 3. Luxembourg: Office forOfficial Publications of the European Communities.

Fetterman, D. M., S. J. Kaftarian and A. Wandersman (1996) Empowerment Evaluation:Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks, CA: Sage.

Floc’hlay, B. and E. Plottu (1998) ‘Democratic Evaluation: From Empowerment Evalu-ation to Public Decision-Making’, Evaluation 4(3): 261–77.

Friedland, R. and R. R. Alford (1991) ‘Bringing Society Back In: Symbols, Practices, andInstitutional Contradictions’, in W. W. Powell and P. J. DiMaggio (eds) The New Insti-tutionalism in Organizational Analysis. Chicago, IL: University of Chicago Press.

Gray, A. and B. Jenkins (1995) ‘Policy Evaluation in a Time of Fiscal Stress: Some Reflec-tions from British Experience’, in R. C. Rist (ed.) Policy Evaluation: Linking Theory toPractice. Aldershot: Edward Elgar.

Evaluation 6(4)

452


Guba, E. G. and Y. S. Lincoln (1989) Fourth Generation Evaluation. Newbury Park, CA:Sage.

Habermas, J. (1984) The Theory of Communicative Action. Volume 1: Reason and theRationalization of Society. Boston, MA: Beacon Press.

Harvey, D. L. and M. Reed (1996) ‘Social Science as the Study of Complex Systems’, in L.D. Kiel and E. Elliot (eds) Chaos Theory in the Social Sciences: Foundations and Appli-cations. Ann Arbor, MI: University of Michigan Press.

Hasluck, C. (2000) Early Lessons from the Evaluation of the New Deal Programmes, ESR49. Sheffield: Employment Service.

Hellstern, G.-M. (1986) ‘Assessing Evaluation Research’, in F.-X. Kaufmann, G. Majone,V. Ostrom and W. Wirth (eds) Guidance, Control and Evaluation in the Public Sector.Berlin: Walter de Gruyter.

Henkel, M. (1991) Government, Evaluation and Change. London: Jessica Kingsley.Hill, M. (ed.) (1993) New Agendas in the Study of the Policy Process. London: Harvester

Wheatsheaf.Hjern, B. and C. Hull (1982) ‘Implementation Research as Empirical Constitutionalism’,

European Journal of Political Research 10: 105–15.Hjern, B. and D. O. Porter (1997) ‘Implementation Structures: A New Unit of Adminis-

trative Analysis’, in M. Hill (ed.) The Policy Process: A Reader. 2nd edn. London: Pren-tice-Hall/Harvester Wheatsheaf.

Hutton, W. (1999) ‘Throwing Good Money after Bad Lads’, Observer newspaper, 17January.

Jenkins-Smith, H. and P. A. Sabatier (1993) ‘The Study of Public Policy Processes’, in P.A. Sabatier and H. Jenkins-Smith (eds) Policy Change and Learning: An AdvocacyCoalition Approach. Boulder, CO: Westview Press.

Julnes, G., M. M. Mark and G. T. Henry (1998) ‘Promoting Realism in Evaluation: Real-istic Evaluation and the Broader Context’, Evaluation 4(4): 483–504.

Kaufmann, F.-X. (1986) ‘The Relationship Between Guidance, Control and Evaluation’,in F.-X. Kaufmann, G. Majone, V. Ostrom and W. Wirth (eds) Guidance, Control andEvaluation in the Public Sector. Berlin: Walter de Gruyter.

Kettunen, P. (1994) Implementation in a Multi-Organizational Setting: Local Networks inEnvironmental Health Policy. Annales Universitatis Turkuensis. Turku, Finland: TurkunYliopisto.

Knox, C. (1995) ‘Concept Mapping in Policy Evaluation: A Research Review of Com-munity Relations in Northern Ireland’, Evaluation 1(1): 65–79.

Kogan, M. (1999) ‘The Impact of Research on Policy’, in F. Coffield (ed.) Speaking Truthto Power: Research and Policy on Lifelong Learning. Bristol: Policy Press.

Laughlin, R. and J. Broadbent (1996) ‘Redesigning Fourth Generation Evaluation: AnEvaluation Model for the Public-Sector Reforms in the UK?’, Evaluation 2(4): 431–51.

Leeuw, F. (1995) ‘Policy Theories, Knowledge Utilization and Evaluation’, in R. C. Rist(ed.) Policy Evaluation: Linking Theory to Practice. Aldershot: Edward Elgar.

Lowndes, V. (1996) ‘Varieties of New Institutionalism: A Critical Appraisal’, PublicAdministration 74: 181–97.

March, J. G. and J. P. Olsen (1984) ‘The New Institutionalism: Organizational Factors inPolitical Life’, American Political Science Review 78: 734–49.

March, J. G. and J. P. Olsen (1989) Rediscovering Institutions: The Organizational Basis ofPolitics. New York: The Free Press.

Marion, R. (1999) The Edge of Organization: Chaos and Complexity Theories of FormalSocial Systems. Thousand Oaks, CA: Sage.


453


Martin, S. and I. Sanderson (1999) ‘Evaluating Public Policy Experiments: Measuring Out-comes, Monitoring Progress or Managing Pilots?’, Evaluation 5(3): 245–58.

Ormerod, P. (1998) Butterfly Economics: A New General Theory of Social and EconomicBehaviour. London: Faber and Faber.

Parsons, W. (1995) Public Policy: An Introduction to the Theory and Practice of PolicyAnalysis. Cheltenham: Edward Elgar.

Patton, M. Q. (1986) Utilization-Focused Evaluation. Newbury Park, CA: Sage.Pawson, R. and N. Tilley (1997) Realistic Evaluation. London: Sage.Reed, M. and D. L. Harvey (1992) ‘The New Science and the Old: Complexity and Realism

in the Social Sciences’, Journal for the Theory of Social Behaviour 22(4): 353–80.Rossi, P. H. and H. E. Freeman (1993) Evaluation: A Systematic Approach, 5th edn.

Newbury Park, CA: Sage.Sabatier, P. A. (1986) ‘What Can We Learn from Implementation Studies?’, in F.-X. Kauf-

mann, G. Majone, V. Ostrom and W. Wirth (eds) Guidance, Control and Evaluation inthe Public Sector. Berlin: Walter de Gruyter.

Sabatier, P. A. (1997) ‘Top-Down and Bottom-Up Approaches to ImplementationResearch’, in M. Hill (ed.) The Policy Process: A Reader, 2nd edn. London: Prentice-Hall/Harvester Wheatsheaf.

Sanderson I. (2000) ‘Evaluating the Effectiveness of Policy Responses to Social Exclusion’,in J. Percy-Smith (ed.) From Exclusion to Inclusion: Policy Responses to Social Exclu-sion in the UK. Buckingham: Open University Press.

Schmid, G. (1996) ‘Process Evaluation: Policy Formation and Implementation’, in G.Schmid, J. O’Reilly and K. Schomann (eds) International Handbook of Labour MarketPolicy and Evaluation. Cheltenham: Edward Elgar.

Schmid, G. (1997) ‘The Evaluation of Labour Market Policy: Notes on the State of theArt’, Evaluation 3(4): 409–34.

Schmid, G., J. O’Reilly and K. Schomann (eds) (1996) International Handbook of LabourMarket Policy and Evaluation. Cheltenham: Edward Elgar.

Schwandt, T. A. (1997) ‘Evaluation as Practical Hermeneutics’, Evaluation 3(1): 69–83.Shaw, I. (1999) Qualitative Evaluation. London: Sage.Thompson, P. and D. McHugh (1995) Work Organizations: A Critical Introduction, 2nd

edn. Basingstoke: Macmillan.Van der Knaap, P. (1995) ‘Policy Evaluation and Learning: Feedback, Enlightenment or

Argumentation?’, Evaluation 1(2): 189–216.Vanderplaat, M. (1995) ‘Beyond Technique: Issues in Evaluating for Empowerment’,

Evaluation 1(1): 81–96.Weiss, C. H. (1995) ‘The Haphazard Connection: Social Science and Public Policy’, Inter-

national Journal of Educational Research 23(2): 137–50.

IAN SANDERSON is Professor of Public Policy Analysis in the PolicyResearch Institute, Leeds Business School, UK. His research interests are focusedon policy and programme evaluation, especially in the fields of labour markets,regeneration and social exclusion, and performance management in localgovernment. Please address correspondence to: Policy Research Institute, LeedsMetropolitan University, Bronte Hall, Beckett Park Campus, Leeds LS6 3QS.[email: [email protected]]

Evaluation 6(4)

454


Date post:	14-Apr-2018
Category:	Documents
Upload:	vodan
View:	219 times
Download:	6 times

Evaluation in Complex Policy Systems - ULiege Evaluation in Complex Policy Systems 435 05 Sanderson...

Documents