1

What Kind of Theory Should Drive Theory-driven Evaluation?

Program Logics and Realist Program Theory in the Evaluation of a Dairy Extension Program

G ElsworthG DrysdaleD GoodrickM Walton

C.I.R.C.L.E

2

Target 10A state-wide dairy industry development program in Victoria, AustraliaInvolves both government and industry agenciesPrimary aim is to improve the productivity and sustainability ofdairy farm businesses and their role in rural communitiesCommenced in 1992• Early focus on improving grazing management• Later extended to other feed-base management programs• Capacity building programs in business planning also introduced• Later emphasis on developing farmer capacity to adapt to

change by encouraging a ‘learner-centered’ culture in the industry

• Evaluation framed by Bennett’s outcomes hierarchy

3

A Typical Representation of Bennett’s Outcomes Hierarchy

(From ‘Targeting Outcomes of Programs (TOP)’ http://deal.unl.edu/TOP/english/contentsf.html)

4

Outcomes HierarchyTypically, a ordered set of component program outputs and outcomes that must be attained before achievement of a longer-term outcome is possibleIn Educational Psychology an early influential outcomes hierarchy was Gagne’s Learning Hierarchy• Learning tasks can be organised according to complexity• Pre-requisite lower-order learning tasks need to be

completed to facilitate attainment of higher-order tasks• Served as the basis for sequencing instruction and led to

the idea of ‘chaining’ – the connection of a set of Ss R components into a sequence

Program outcomes hierarchies appear to be based on very similar principles

5

Is an Outcomes Hierarchy an Appropriate Theory for

Theory-driven Evaluation?

6

Program Theory in EvaluationHistorical Overview

Suchman (1967)• Two reasons for program failure: implementation failure; theory

failureWeiss (1972)• Introduced the idea that an evaluation could be based on a

process theory of the programWholey (1979, 1983)• Elucidation of program theory as part of evaluability assessment:

Prior to the start of a formal evaluation, analyse the program logicto ascertain whether the program, as designed, should be able tobring about the desired outcomes

Chen & Rossi (Chen, 1990)• ‘Theory-driven evaluations’: an evaluation should be based on a

prior explication of the program theory, which is then tested by the evaluation study. This should be a social science theory, not simply a series of ad hoc logical premises about how the programelements fit together (‘program logic’). Importantly, it contains an account of the mechanisms that bring about program effects

7

Program Theory in Theory-driven Evaluation

Two IssuesWhat is an appropriate epistemological grounding for a program theory in Theory-driven Evaluation?

What evaluative purpose should a program theory serve in Theory-driven Evaluation?

8

Epistemological Grounding

Realism vs Empiricism

9

Situating Scientific Realism

result/regularity

model-building

transcendental idealismempirical-testingreal

(3)

events; sequences; invariances

imagined/imaginary

(1) classical empiricism

(2)

transcendental realism

generative mechanisms in models

---------------------------------------------------------------------

---------------------------------

[Taken from Bhaskar (1978) 145]

The Logic of Scientific Discovery

10

‘Scientific’ (or ‘Transcendental’or ‘Critical’) Realism

Two significant differences when realism is

compared with the ‘classical’ empiricist

(‘positivist’) paradigm of program evaluation

• A radically different model of causal explanation

• A radically different view of the relationship between

‘facts’ and ‘values’

11

Different Models of Causal ExplanationClassical Empiricism

‘Covering law’ from which the facts or (subsidiary) laws to be explained can be deduced

‘Constant conjunction of events’ is both necessary and sufficient evidence to establish a causal relationship

Constant conjunctions occur within ‘closed systems’

12

Arguably Results in a Limiting Model of Causality in Complex Social Programs

Causal phenomena in the human domain of social programs are

More complexMultiply determinedMultiply determiningNon-linearConditional on specific contextsFrequently, more radically unanticipated

Than a simplified ‘closed system’ model can adequately portray

13

Different Models of Causal ExplanationScientific Realism

Begins with an ontological assumption - the phenomenon to be explained are the overt manifestations of the activation (‘triggering’) of an (unobserved) generative structure (Fararo, 1989, p. 39) – that is, reality is stratifiedObservation of a constant conjunction of events is neither necessary or sufficient to establish causalityIn an open system, observed outcomes are radically context dependent, such that a mechanism, if triggered, will only result in observable change in the appropriate contextCausal explanation typically involves a hypothetical model of the structure, built by analogy with a known structure (the positionadvocated, particularly, by Harré – Fararo, 1989, p. 40)

14

The ‘Ontological Assumption’ of an Embedded Reality – Bhaskar’s Model

Real Actual Empirical

Structures ×

Events × ×

Experiences × × ×

15

The ‘Ontological Assumption’ of an Embedded Reality – Harré’s Model

Realm 1 - entities that are available to immediate sensory observationRealm 2 - entities that are potentially available to direct sensory observationRealm 3 - entities that are real but will never be available to sensory observation (entities that can be known, but might never be shown to exist)Realist explanations of social events frequently entail entities that appear to be in Harré’s Realm 3 (structures and mechanisms that are inferred from experiences but cannot be directly observed)

16

Thus Pawson & Tilley (1997, pp. 68-72) Contrast Various ‘Successionist’ (Empiricist)

Models of CausalityX Y

X

Z

Y

X

Z

Y

A directsuccessionist model

A model in whichZ ‘mediates’ the relationshipbetween X and Y

A successionist modelsuggesting thatX and Y are‘spuriously’ causedby Z

(Pawson, R., & Tilley, N. (1977). Realistic Evaluation. London: Sage. p. 68)

17

With a Model in Which a Program is Viewed as ‘Triggering’ the Action of a Generative Structure

An action is only causal if it triggers, in Context, a Mechanism that results in the Regularity (outcome).(Pawson, R., & Tilley, N. (1977). Realistic Evaluation. London: Sage. p. 72)

Context

Mechanism

Regularity

18

Is Reconciliation Between Empiricist and Realist Theories Possible?

Fararo (1989) presented a compelling two-step argument for the reconciliation of ‘positivist’ theory with realism in his account of ‘general theoretical sociology’Step 1• Mathematical axiomatics (e.g. Suppes) is embedded within an

instrumemtalist ‘meaning hierarchy’ theory structure (e.g. Toulmin) as a desirable but not necessary attribute of the latter

Step 2• Meaning hierarchy theory structures are construed as being

compatible with realism if the entities proposed by the theory are believed to really exist (i.e. are not simply regarded as usefulfictions)

19

Evaluative Purpose

Normative vs Causal Theory

20

Program TheoryH. T. Chen’s Definition

(A program theory is) “a specification of what must be done to achieve the desired goals (of the proposed program, or the program to be evaluated) what other important impacts may also be anticipated, and how these goals and impacts would be generated.”

(Chen, H. T. (1990). Theory-driven Evaluations. Thousand Oaks, CA: Sage. p.43)

21

Thus Chen Proposed Two Kinds of Program Theory

Normative (Prescriptive) Program Theory: “a specification of what must be done to achieve the desired goals”

Causative (Descriptive) Program Theory: “a specification … of how these desired goals would be generated”

22

The Three ‘Domain Theories’ of Normative Program Theory

Treatment Theory• Specifies what the nature of the program

treatment should beImplementation Environment Theory• Specifies the nature of the context within

which the program should be implementedOutcome Theory• Specifies what the nature of the program

outcomes should be

23

The Three ‘Domain Theories’ of Causative Program Theory

Impact Theory• Specifies the causal link between the treatment and

the outcomeIntervening Mechanism Theory• Specifies how the underlying intervening processes

operateGeneralisation Theory• Specifies the generalisability of evaluation results to

the topics and circumstances of interest to stakeholders

24

Chen Thus Sees Two Clear-cut Elaborations of Program Theory

‘Purely’ Normative• Where the axioms describe how people should

evaluatively orient to the world

‘Purely’ Empirical• As is a psychological theory where the axioms of the

theory enable the derivation of claims that can be empirically falsified

(The distinction between ‘purely normative’ and ‘purely empirical’ elaborations of a theory was made by Fararo, T. J. (1989). The Meaning of General Theoretical Sociology. Cambridge: Cambridge UIniversity Press. P 20.)

25

This Categorical Distinction Appears to Lead to Two Potential Confusions in Chen’s Structure of Program Theory

26

In Addition to Arguing for an Integration of ‘Positivist’ and Realist Theory Structures,

Fararo (1989) Recognised that ‘Hybrid’ Elaborations of a Theory are Possible

A hybrid elaboration “… recognizes that the axioms might well be falsified in careful empirical investigations, yet it regards them as defining an idealized model of action as a starting point for theoretical explanations in a variety of contexts.”

(Fararo, T. J. (1989). The Meaning of General Theoretical Sociology. Cambridge: Cambridge UIniversity. P. 20.)

27

From an Empiricist Perspective We Have an ‘Outcomes Hierarchy’ Derived From ‘Axioms’ About

the Contingent Relationships Between Events Which Yields

Explicit hypotheses about the temporal ordering of events duringprogram implementation

An initial template for refining specific evaluation questions and designing observations and measurement

A caution that longer-term outcomes may not be attainable (and hence observable) within the duration of the proposed evaluation

And hence, a model for positing that short and medium term outcomes provide some evidence that longer-term outcomes may be achieved

28

From a Realist Perspective We Have

A focus on the causal mechanisms that are triggered or enabled by a programThe proposition that program theories should highlight both micro (individual) and macro (social or community level) mechanisms and the causal relationships and interactions between the two

29

Thus a ‘Normative’ Program Theory Should, Critically, Contain Accounts of

The ways that the reasoning and subsequent choices of the program participants are expected to change as a result of participation in the programThe ways that the capacities and resources of the social group(s) will be developed and enhanced to support these ‘micro-level’ changes

30

And a ‘Causative’ Program Theory Should Contain

accounts of how these mechanismswere actually generated and how they led to the desired program outcomes

31

Conclusion

A Proposed Hybrid Structure for Program Theory

(With Embedded Elements of an Outcomes Hierarchy)

32

A Revised Structure for Program Theory

In what other general contexts does the program succeed?

What other contexts will the program successfully operate in?

A GeneralisationTheory

What causal mechanisms were actually triggered by the program? In which contexts?

How will the program bring about the desired outcomes?

A Theory of Causal Mechanisms

What were the actual implementation contexts?Which were associated with the intended outcomes?

What contexts is the program designed to operate in?

A Theory of the Implementation Environment

What structures /strategies were actually implemented?

What structures /strategies will constitute the program?

A Theory of Intended Treatments

What are the actual outcomes?

What are the desired outcomes?

A Theory of Intended Outcomes

What client needs did the program actually meet?

What client needs is the program designed to meet?

A Theory of NeedCausativeNormative