Case Study Evaluation: Past, Present and FutureChallengesCase Study as Antidote to the LiteralSaville Kushner

Article information:To cite this document: Saville Kushner . "Case Study as Antidote to the Literal" InCase Study Evaluation: Past, Present and Future Challenges. Published online: 13 Jan2015; 63-83.Permanent link to this document:http://dx.doi.org/10.1108/S1474-786320140000015003

Downloaded on: 20 January 2015, At: 13:21 (PT)References: this document contains references to 0 other documents.To copy this document: permissions@emeraldinsight.comAccess to this document was granted through an Emerald subscription provided byToken:BookSeriesAuthor:EA032558-91CB-4134-8A12-EE215C0717AD:

For AuthorsIf you would like to write for this, or any other Emerald publication, then pleaseuse our Emerald for Authors service information about how to choose whichpublication to write for and submission guidelines are available for all. Please visitwww.emeraldinsight.com/authors for more information.

About Emerald www.emeraldinsight.comEmerald is a global publisher linking research and practice to the benefit of society.The company manages a portfolio of more than 290 journals and over 2,350 booksand book series volumes, as well as providing an extensive range of online productsand additional customer resources and services.

Emerald is both COUNTER 4 and TRANSFER compliant. The organization is a partnerof the Committee on Publication Ethics (COPE) and also works with Portico and theLOCKSS initiative for digital archive preservation.

*Related content and download information correct attime of download.

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

CASE STUDY AS ANTIDOTE TO

THE LITERAL

Saville Kushner

ABSTRACT

Much programme and policy evaluation yields to the pressure to reporton the productivity of programmes and is perforce compliant with theconditions of contract. Too often the view of these evaluations is limitedto a literal reading of the analytical challenge. If we are evaluating X welook critically at X1, X2 and X3. There might be cause for embracingadjoining data sources such as W1 and Y1. This ignores frequent reali-ties that an evaluation specification is only an approximate starting pointfor an unpredictable journey into comprehensive understanding; that thespecification represents only that which is wanted by the sponsor, andnot all that may be needed; and that the contractual specification toooften insists on privileging the questions and concerns of a few. Casestudy evaluation proves an alternative that allows for the less-than-literalin the form of analysis of contingencies � how people, phenomena andevents may be related in dynamic ways, how context and action haveonly a blurred dividing line and how what defines the case as a case mayonly emerge late in the study.

Keywords: Case study; evaluation; contingency

Case Study Evaluation: Past, Present and Future Challenges

Advances in Program Evaluation, Volume 15, 63�83

Copyright r 2015 by Emerald Group Publishing Limited

All rights of reproduction in any form reserved

ISSN: 1474-7863/doi:10.1108/S1474-786320140000015003

63

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

EDITORS’ INTRODUCTION

Most case study evaluators will be familiar with the tension addressed inthis chapter. On the one hand, funders and other stakeholders often expecta tightly planned work programme in which hypotheses, data sources, ana-lytic methods and intended outputs are rigidly pre-specified � a situationthat requires to use Kushner’s withering phrase, ‘the narrow specificationof variables and the freezing of context’. On the other hand (arguably), thepurpose of case study is not to define the linear and causal relationshipsbetween a rigidly defined set of variables but to generate a rich, vivid andnuanced account of particular actors and events in context. The formerapproach, which many would conflate with ‘scientific’ or ‘rigorous’ metho-dology, is referred to by Kushner as ‘literalism’. But the latter approach,which Kushner calls ‘contingency’, offers far more promise to generate the-ory and explore complex (i.e. non-linear) causality.

**************************************************

I was at a conference on qualitative approaches to music education.Elliot Eisner made an inspiring presentation on the use of the visualarts as a source of learning for us, educational researchers and eva-luators who tend to confine ourselves to the dry art of formal, textualrepresentation. He gave a number of examples from film andpainting � all of them figurative. Robert Stake raised his hand for aquestion � ‘what does abstract expressionism look like in educationalresearch’?

I want to juxtapose ‘literalism’ in educational evaluation with a focus on‘contingency’. I will come to an explanation shortly � for now, I will give acase example. The case was an evaluation of bilingual education policy inthe United States, embracing a single school study, a study of the commu-nity and local context, of policy history and of politicians (MacDonald &Kushner, 1982). In a broader sense, this was a case study of the contempor-ary civil rights movement in the United States. The specific focus was onthe Rafael Hernandez School, Boston. Set up by the Puerto Rican

64 SAVILLE KUSHNER

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

population and recently incorporated by the School District, the schoolhad been selected for its success and prominence.

There was an interview programme with civil rights activists, politiciansand administrators in Boston, attendance at a Washington policy forumfor bilingual education, a case study of the Hernandez School in its com-munity, ancillary studies of two other schools in Boston and a wide-rangingreview of research literatures and policy documentation at different levelsof the political system. Personal biography featured highly.

This was designed as a policy evaluation and it was a test of how to gen-eralise from a single instance using Democratic Evaluation (MacDonald,1987) and case study to create a collaborative relationship with the schooland to portray its practices and values. The collaboration would theoriseover political, social and educational issues associated with bilingualschooling and the implications for policy. We had no ‘research questions’,no starting focus, few presuppositions, there was no ‘program logic’, noobjectives to measure and no assumed causal link between policy andpractice. If any of these existed we were to discover them empirically.Nonetheless, we had a specialist in sociolinguistics on the team and he wasto conduct classroom observations against the known canon.

We conducted a range of classroom observations underpinned by, butalso demonstrating, prior understanding of sociolinguistic and peadagogi-cal analysis. These formed the centrepiece of a draft case study of theschool. Since we were looking at bilingual teaching and curriculum, wechose to observe classroom teaching practices. As it happens, the schoolrejected this draft and withdrew access. We negotiated out way back in onthe basis that they negotiated the methodological focus � that is, we repre-sented them in their own terms.

But what we had done was to take literally the brief to apprehend theessence of bilingual education � we wanted to concentrate on bilingualpedagogy. Here is a typical extract from the draft case study report thatwas rejected:

Grade 1 � English Reading[…]

‘Literacy is premised on a knowledge of the cultural componentsthat the terms, sentences, paragraphs and texts refer to. Literacy inEnglish requires that the Spanish monolingual and dominants fromHispanic homes learn about the referents of the anglo culture. English

65Case Study as Antidote to the Literal

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

literacy � understanding the cultural significance of what is read � isvital for academic achievement in an English-dominant state. In this les-son the beginnings of literacy, learning to read, are being taught at thebeginning of the new school year.

Wendy asks the class to recite the sounds of the English vowels so thatshe may hear them. As they recite, Wendy points with a stick to the let-ters on display. Each of these letters has a picture of an object in whichthe vowel sound is made � long ‘a’ as in ‘cake’, long ‘o’ as in ‘rose’,short ‘o’ as in ‘top’’.

[…]

Teacher: Can you repeat that? Long ‘o’ says ‘ø’’ � try it. Long ‘o’ says‘ø’, just like its name. Okay? Do you know the name of this letter inEnglish? That’s ‘a’. Long ‘a’ says a. What is it? Long ‘a’ says a. Okay?This is ‘e’, the name of this letter in English. Long ‘e’ says ‘e’. Say it.

Pupil: Long ‘e’ says ‘e’.

Teacher: Good. Okay. This is ‘i’….

[…]

The sound/letter correspondence section concludes. Reading wholewords and sentences commences. These words incorporate the lettersand sounds of the parts of this and preceding lessons.

And here is an example of how we analysed this kind of data:

By pointing to pictures of objects and getting the children to recite the appropriate

vowel sounds the teacher helps the children to memorise the sound that matches the

word that fits the picture. Although the formulation of the rule (e.g. in the form long

‘o’ says ‘ø’, long ‘u’ says ‘uu’) might be memorised, the application of the rule is depen-

dent upon the student’s recognition of a word to which the rule applied, and the lear-

ner’s ability to reproduce the sounds that are appropriate in that language. …The

acquisition of the connection between sound and letter-symbols is manifest in the

appropriate utterance of the sound….This approach to the teaching of reading through

initiation into sound/letter relations is known as the Phonic method. A main alternative

method in use is usually called ‘Look and Say’. The sound/letter relationships in Look

and Say are acquired inductively…’

There were strong objections from Wendy and her colleagues whoalleged that we were not representing their efforts and their quality in a

66 SAVILLE KUSHNER

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

way that was fair or accurate. What they noticed were two things: first,that we took the task too literally. Our classroom observations werefocused on a formal curriculum which they were expected to teach � notone they preferred to teach and which represented their view that cultural(not linguistic) principles underpinned good bilingual education:

But where do you see [culture]? How, when I am teaching English reading, am I going

to have 90% of that lesson in ‘culture’? Come to culture classes, okay, and you’d see it.

Come to the Music class, see it … they’re using a Spanish instrument � they’re not

using an American instrument…

The second objection was that we had asserted evaluator authority overthe interpretation of the case, and that we had drawn from formal, not situ-ated theory. Look back at the first paragraph of the observation and at theparagraph showing the data analysis.

They were surprised � so were we. They saw themselves as bilingual edu-cators; we were portraying them as bilingual teachers. We were ejectedfrom the school. To regain access we agreed to allow them to shape themethodology, to be represented in their own terms. What they insistedupon was that we observed their cultural practices (school picnic, sportsday, honours day) and generate a situated theory of how they went aboutthings. Here is an extract from a classroom observation that came out ofthis second phase � it was preceded by a short biography of another tea-cher, Sheila, which portrayed her as a woman who was determined andunafraid to shape her own life and practices:

Sheila described to us another curriculum problem:

‘At one point this year, I was almost going to do Social Studies threedays in English and two days in Spanish … but you’ve got to rememberthat my kids, even my Hispanic kids, except for two, are fairly good inEnglish at this point, and I watered the concept down into their simpleEnglish understanding…

Sheila goes on to the next question which is about the explorers, Shereads from the text and mentions the London Virginia Company.“What’s a company?” she asks. Mike replies, “Like the Boston HeraldAmerica.” “How do you know it’s a company?” Sheila asks. Mikereplies, “I do.” The word ‘company’ is translated by and for some ofthe Hispanic pupils’.

[…]

67Case Study as Antidote to the Literal

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

If there is actual watering down of the curriculum Sheila believes itrelates not only to language problems but also to the availability oftime. The conversation develops into specific problems of bilingualpedagogy:

‘Okay, but the problem with teaching Social Studies here is thelanguage…’?

No, no. The problem with teaching Social Studies is that we’re dealingwith two languages and it’s taking time…

[…]

But what’s the difference between bilingual teaching and teaching inEnglish and Spanish alternately the same lesson? Or is there none � isthat what bilingual teaching is?

I don’t know. See, to me it’s very hard to describe bilingual teaching,because every year my bilingual teaching changes.

Does it? Can you give me an example?

It has to change. This year I have nine speakers of English out of aclass of 25….

[…]

As this observation develops, Sheila enunciates certain principles thatunderpin her teaching � ‘watering down’ emerges as just one. From theseand other data we were able to develop a situated theory of bilingual peda-gogy � and, once we had documented the life and relationships in theschool, a theory of bilingual curriculum � both contextualised in thisschool and its particular practices, but available for generalisation where itfound purchase on thinking elsewhere.

There were many things happening here. Guided by the school itself, wedevised a practical ethic which informed an emergent case study design.Abandoning what I will call a ‘literal’ approach to the evaluation casestudy (‘this is a case study of bilingual schooling, therefore, we will focuson bilingual teaching’) brought into question the intellectual authority ofthe evaluators as well as our authority over the design. We were forced toenter into the values domain of the school in order to shift from applyingsociolinguistic theory in favour of theorising with teachers about how to

68 SAVILLE KUSHNER

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

explain their practices. In fact, this methodological shift distinguished whatstarted as a research exercise (enquiry motivated by interests and theoriesindependent of those who inhabit the case) to an evaluation exercise(enquiry shaped by the people in the case who live the consequences).

More importantly, however, whereas the former approach prescribed acertain form of ‘codified’ knowledge as a medium through which to under-stand the quality of bilingual education, what the modified approachallowed for was an acknowledgement that all human projects are charac-terised by a confluence of multiple forms of knowing and that understand-ing action and organisation requires their capture (Russell et al., 2004). Inthe second observation, we were combining Sheila’s experience with thefacilitation of her tacit pedagogical knowledge and direct observation of asituation.

THE LITERAL AND THE CONTINGENT

Case study provides us with an opportunity to escape from a restrictive ‘lit-eralism’ which constrains ways of knowing, and a way into the more uncer-tain realm of ‘contingency’, in which multiple knowledge forms can beapprehended. Rather than examine how action conforms (or otherwise) toa pre-determined rationality, contingency requires us to search for andreconstruct those often diverse forms of rationality that are at play in thecase. There will be more than one (no programme has a single ‘logic’). Ifour proper concern is less with what works than with how do things work(Stake, 2010), both literalism and contingent analysis address this, but inways that arise out of different understandings of knowledge and distinctways of seeing the world.

Literalism: adherence to an explicit or a given idea or proposition.Methodologically, this places interpretive authority in the hands of the eva-luator who asks the question.

Contingency: that an idea or a proposition finds its meaning throughdependence on other factors, as yet unspecified. This most often will trans-fer authority to the evaluands on whom the evaluator relies to explaincoherence.

‘Literalism’ finds a place in the study of religious texts, where the ques-tion is whether a moral text � even a divine utterance � is to be taken atface value or interpreted. Besecke (2007) argues that literal readings fore-close on possibilities of a ‘reflexive spirituality’ � that is, that it is

69Case Study as Antidote to the Literal

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

important for people to be able to interpret moral texts in order to developpersonal principles through interacting with them, and that literalism dis-places reflection. It finds a place, again, in legal and constitutional dis-course where ‘literalism’ too easily gives rise to the absurd � such as whena law prevents the shedding of blood ‘in the streets’, which would deny asurgeon the capacity to conduct an emergency operation (Dougherty,1994). And it finds yet another place in theories of art and architecture,where the literal stands for the ‘figurative’ � the direct representation of arecognisable phenomenon (Linder, 2005) and juxtaposed with ‘the abstract’in representation.

My use of the term ‘literal’ has echoes of each of these. The authoritativerepresentation of Wendy’s class (above) forecloses on the interpretativerole of the reader (in semiotic terms this is a ‘readerly text’ � that is, one inwhich both writer and reader are trapped by singular narrative meanings).In parallel with a ‘reflexive spirituality’, what we are after with Sheila’s textis a ‘reflexive pedagogy’, one that benefits from situation interpretation �hers is a ‘writerly text’, one in which the writer gives readers an interpretiveresource (not a determinant).

In terms of a legal/constitutional critique of literalism, evaluativeenquiry has to represent the world as implied in their contracts but has alsoto take on board the potential distortions caused by an exclusive allegianceto contract. Dougherty counterposes ‘absurdity’ with ‘literalism’, but notas a dualism � she argues that a recognition of absurd outcomes of the lit-eral view is a useful discipline to the literal � that it is complementary toliteral readings. It is impossible for legislators (pace contract writers) toforesee all eventualities, so some interpretive flexibility has to be assumedin law for the spirit of a constitutional arrangement (e.g. ‘rule of law’) tomaintain its hegemony. Similarly, were our portrayal of Wendy’s class tohave been the core of our case study, we would have portrayed bilingualeducation merely as a technology, immune from human interactivemeaning.

The art/architecture controversy over literalism also has its useful learn-ing. One view in this discourse is that recognisability is important for pur-poses of identification, but that what is figurative can nonetheless beadapted to provoke pluralist responses and interpretations. There is, ineffect, a spectrum from ‘literal’ to ‘abstract’, and representations fall some-where along it. All dualisms collapse empirically, and it is well to be awarethat evaluators can embrace no extremes. What is literal (taking the ques-tion as given) does not preclude interpretive readings � so long as literalrepresentations are inviting of the reader’s scepticism.

70 SAVILLE KUSHNER

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

What I want to get out of these, however, and of the juxtaposition ofthe two classroom observations is that ‘literal’ and ‘contingent’ are, if not adualism, underpinned by distinct epistemologies. Literalism assumes thatthere is a world of action that can be pre-figured in the form of questionsthat have a prior relevance. This assumes, too, that the questions that pre-occupy an evaluation sponsor or other stakeholder stand for reasonablepreoccupations � that is, reasonable in the sense that they correspond withreal possibilities for resolution. There is a presupposed stability to thisworld, such as exhibits a linear rationality from intent to action, from ques-tion to answer, and from problem to solution.

Clearly, my argument is that such stability is illusive, and that we mustgive way to non-linear rationality � to worlds in which relationships aredynamic and interacting rather than determining. It may be possible �though unlikely (as we will see below) � that we could pin down enoughvariables to explain an effect and cancel out situational ‘secondary’ effects.But even that would fail to explain, since we would then need to gofurther and pin down the interactive nature of mutually determining rela-tionships. We might pin down a generalisation about how school leader-ship improves the quality of schooling, with a thoroughgoing study ofsocial, political, cultural, psychological, social-psychological, economic,institutional, pedagogical, professional and experiential variables (is itworth the effort?) � but we would still need to document how a schoolleader’s interactions with teachers, ancillary staff and students, parentsand their own religious, say, commitments helped shape her viewsand practices (see Wolcott, 1973 for just such a case study). Contingencyis all.

CASE STUDY AS THE STUDY OF CONTINGENCY

I am thinking of how we approach the object of our evaluations and whywe choose case study to look at it. Case study grew out of dissatisfactionwith surface realities and the failure of social enquiry to penetrate them(Jocher, 1928), and a realisation that the question you ask in evaluation is,at best, only an approximation to a question worth asking (Kushner,2000). For the literal evaluator pursuing the question as given, the burdenof outcomes is an answer. For the evaluator opting for contingent analysisthrough case study, emerging with the right question to ask is a valid end.Shaw (1927):

71Case Study as Antidote to the Literal

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

Case study method emphasizes the total situation or combination of factors, the

description of the process or sequence of events in which the behavior occurs, the study

of individual behavior in its total social setting, and comparison of cases leading to for-

mulation of hypotheses.

I would say that literalist enquiry is designed to respond to what the eva-luation sponsor wants; contingent enquiry is more likely to yield informa-tion that the sponsor and others need. The raising of hypotheses is a morevalid evaluation result than finding answers that may only serve to displacepolicy deliberation. All policies are hypotheses to be tested in the labora-tory of social action, and an obligation of evaluation as a service is to helpframe those hypotheses. Case study is the principal methodology forevaluation.

We may pursue an idea or an instruction as it is given, assuming it corre-sponds well with its intent, and that it is self-sufficient. We want to assessthe effectiveness of a ‘synthetic phonics’ initiative in teaching reading � weadminister pre- and post-reading tests to young children and we ask tea-chers about their professional view. We need to make judgements aboutthe implementation of nurse competence in trauma settings so we set up anobservation schedule based on agreed and trained competence criteria,backed up with nurse and doctor judgement and, perhaps with patient per-spectives. We can elaborate these designs with mixed methods, studies ofcontextual variation, mixed sampling techniques and so on. We aim toemerge with results that echo � within reasonable limits � our initialspecifications.

Something that is literal is pre-specified, precise and self-defining.Something that is contingent has boundaries discovered situationally, andit relies for its sufficiency on dynamic relationships with other phenomena.The literal study distinguishes between action and context (foreground andbackground) and can be sure about which is which.

Something that is literal has its coherence built-in or assumed; this iswhat defines ‘strong’ or ‘robust’ designs � that is, designs for evaluationenquiry that endure the push-and-pull of field experience. Evaluation thatis contingency-focused has to discover what makes it coherent � suchdesigns are not ‘weak’, they are emergent or subject to ‘progressive focus-ing’ (Parlett & Hamilton, 1972). So, your bicycle is literal; your means oftransport is contingent. Your wedding is literal, your marriage, contingent.A programme objective is literal, its meaning for action, contingent.

It is at least prudent to take an idea or an instruction to be just anapproximation to the intent and the meaning behind it, or as a startingpoint to understanding the idea and instruction itself. To assess the

72 SAVILLE KUSHNER

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

effectiveness of synthetic phonics is likely to require an understanding of itsalternatives � in fact, its competitors in a field of some controversy. Oncethe pedagogical complexity of the strategy is established, it will be neces-sary to look at the conditions of its implementation. We must live with theuncertainty that, in a given situation, the implementation of synthetic pho-nics is a pretext, and that something else is happening that gives it authenticmeaning beyond its immediate sponsorship. Maybe what it is really aboutis a bid for control of curriculum or teachers; perhaps the most significantaspect of its implementation is school relationships with non-English lan-guage groups; perhaps the major implication of the implementation is aresponse to the international PISA tests; perhaps synthetic phonics is bestdefined, not as a reading scheme but as a pedagogical relationship.

Similarly, with the case of the trauma nurse and his competencies. Whatare we actually looking at when we observe the nurse in action? We maywell be looking at discrete components of action being operationalised. Butwe might also be looking at a power struggle between nurses and doctorsfor clinical authority � or between trauma care and secondary care; wemay be looking at gender issues in health roles; we may be looking at thepolitical relationship in acute care between the general nursing required intrauma care and specialised nursing required in single-discipline medicine(orthopaedics, gynaecology, oncology); and we (the ‘independent evalua-tors’) may have been contracted by the hospital management to deliver areport that justifies an executive decision to reduce the number of nursesemployed. In both of these cases it is a matter of judgement (not pre-judgement) as to which is contingent on what. Often, this will be given byan understanding of what political or institutional intent lies behind thecontract.

What is literal is more easily specified and controlled, and assumes astability of conditions � it generally lends more certainty, even predict-ability. What is contingent demands to be understood before control canbe considered � it demands tolerance of uncertainty by degrees. When welook at contingency we lose the simple dichotomy between action andcontext, foreground and background. Are we to focus on the nursing com-petencies as action and the politics of acute care as context � or vice versa?We resolve this by developing a ‘theory of significance’ that allows us toorder and prioritise certain kinds of relationship, looking for patterns,offering interpretations of coherence until we discover them � decidingwhich is foreground and which is background, using ‘progressive focusing’to establish case boundaries. And all has to be negotiated. After all, we aretrying to interpret the sense of authenticity of others, not our own.

73Case Study as Antidote to the Literal

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

Causal inference (‘X gave rise to Y’ � an explanation of how thingscome about) can be arrived at through both routes � through literal fram-ing or contingent analysis, these are options � but where it is defined lit-erally it requires the narrow specification of variables and the freezing ofcontext. In my argument, this makes all such causal explanations provi-sional upon testing against judgement and experience. First work out in lit-eral terms what causality consists of (this is research based ondemonstration) then explore its meaning, feasibility, broader implicationsand so on (this is the evaluative element based on judgement). Causal infer-ence arrived at through analysis of contingencies � of dynamic relation-ships � and progressive focusing invites consideration of all availablevariables to enhance the credibility of the analysis.

It follows from these that literalism tends towards the general and thetheoretical; contingency tends towards the particular and the practical.Contingency requires that we understand local significance.

Donald Campbell (1999, p. 115) wrote:

…local could … be taken as implying no generalisation at all, conceptualising a validly

demonstrated cause-effect relationship that we do not yet know how to generalise. The

causal relationship would be known locally … but there would be no validated theory

of it that would guide generalisation to other interventions, measures or populations,

settings or times. This is, of course, an exaggeration. The theories and hunches used by

those who put the therapeutic package together must be regarded as corroborated,

however tentatively, if there is an effect of local … validity in the expected direction.

Nonetheless, this exaggeration may serve to remind us that very frequently in physical

science (and probably in social science as well) causal puzzles (that is, effects that are

dependable but not understood) are the driving motor for new and productive theoris-

ing. We must back up from the current overemphasis on theory first. Basic scientists

put a premium on clarity of causal inference and hence, limit, trim and change pro-

blems so that they can be solved with scientific precision given the current state of the

art. Other causal hypotheses are postponed until the state of the art and theory develop-

ment make them precisely testable. This strategy is not available to applied scientists.

They should stay within the mandated problem, doing the best they can to achieve

scientific validity. To stay with the problem, however, it may be necessary to use meth-

ods providing less precision of causal inference.

Campbell goes on to state a preference for attending to validity (as plau-sibility) more than to clarifying causality � he is reluctant to ‘retreat’ tonarrowly specified variables, content to leave the determination of causalityas a local (case-specific) judgement and generalisation is methodologicallyopportunistic � we do it where and how we can. In respect of our argu-ment here, literalism is the focused pursuit of a restricted range of variableswhich may reveal a generalisable causal chain but will undermine its own

74 SAVILLE KUSHNER

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

credibility and the strength of its generalisability. There is a tensionbetween specification and generalisation. The closer we can identify ‘whatworks’, the less generalisable it will be. Campbell’s point might be reducedto the simple aphorism, ‘do what you can to make sense to people in a par-ticular place, but don’t lose your credibility’.

We hear the same point made by Cronbach (1975, pp. 124�126):

Instead of making generalization the ruling consideration in our research, I suggest that

we reverse our priorities. An observer collecting data in one particular situation is in a

position to appraise a practice or proposition in that setting, observing effects in con-

text. In trying to describe and account for what happened, he will give attention to

whatever variables were controlled, but he will give equally careful attention to uncon-

trolled conditions, to personal characteristics, and to events that occurred during treatment

and measurement. As he goes from situation to situation, his first task is to describe and

interpret the effect anew in each locale, perhaps taking into account factors unique to

that locale of series of events (cf. Geertz, 1973, chap. 1, on “thick description”). As

results accumulate, a person who seeks understanding will do his best to trace how the

uncontrolled factors could have caused local departures from the modal effect. That is,

generalization comes late, and the exception is taken as seriously as the rule ….A general

statement can be highly accurate only if it specifies interactive effects that it takes a

large amount of data to pin down’. [my emphasis]

Campbell was alluding to his seminal nine ‘threats to validity’. Whatearly evaluation case study theorists did was to focus on a 10th ‘threat tovalidity’ implicit in both Campbell’s and Cronbach’s accounts: failing togain purchase on the ‘local’ world of practice. To resolve this dilemmabetween specifying causality and generalising across contexts, they designedcase study to ‘give equally careful attention to uncontrolled conditions’. Theydeparted from the literal and sought a methodology that was likely to yieldwhat we can call a ‘theory of contingency’. How people and events arerelated to each other dynamically (i.e. how each ‘feels’ the feedback of theother) is a more accurate route to understanding how systems work thanclose measurement of small parts of systems.

Evaluation contracts most often tend towards the literal � a pre-specification of the knowledge field that is to be represented. An expecta-tion that what is of concern is what is of concern, and that the evaluationshould waste no time in getting down to it � even though there may beother political agendas in the background. Much flows from this. We arewedded to what we think of as ‘logic models’ because what is regarded inliteral terms as an evaluation task is thought to be sufficiently definableand stable over time and context to be branded with a singular logical set.Synthetic phonics has a logic of action (a theory of change/learning) thatstays still long enough for us to get the measure of it. If we want to

75Case Study as Antidote to the Literal

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

understand synthetic phonics we look at synthetic phonics. If we want toevaluate nursing competencies we observe nursing practice. A given goalwith a pre-specified outcome is assumed to have the potential for a causallink; an outcome is taken to be an indicator of a theory of change in opera-tion; a programme specified by its objectives is expected to behave as thatprogramme; as with a Victorian toy, certain successful machinations aresupposed to make things ‘work’. We may report on these things and missthe point.

In evaluation, what is literal always assumes some degree of linearrationality, forward progress from a relatively undesirable state to a moredesired one, and enough stability of context to allow for a constitutive rela-tionship between an independent and a dependent variable.

The study of contingency allows for each of these conditions to be brea-ched. It may be thought that the relationship between a dependent and anindependent variable is a contingent relationship, but it is not, in the sensethat contingency implies a dynamic relationship based on feedback � thatis, each variable affects the other. This violates linearity since it may reverseit. An example would be as follows: we discover, through observation-based interviewing, that trauma nurses routinely conduct their practicebeyond the reach of the competencies which now appear designed to limitclinical nursing procedures to weight clinical judgement in favour of doc-tors. Nurses complain that they want and feel capable of doing more thanis required of them. We turn back to put the competence policy under criti-cal scrutiny and to identify the power relations underpinning its theory.Which is the dependent variable � the training procedure, the prejudice orthe policy? Which is ‘context’, which is ‘foreground’?

From Campbell and Cronbach we would take the following lessonswhich refer us back to the pluralism of knowledge forms (Russell et al., opcit.):

1. We may assume a linear, causal relationship between training in compe-tencies and nursing outcomes on the basis of ‘literalist’ observationalstudies: ‘mastery of competency #13b gives rise to patient outcome X’.This will be transferable to other settings as a guide for nurse trainersthere, (‘competences work’). But the generalisation will almost certainlybe weak, since the contextual data that made both the conception andthe operationalisation of competencies meaningful is missing � that is,that experience cannot easily be replicated in another hospital. To do sowould require a ‘large amount of data’ to pin down the learning/practiceinteractions � in fact, data from both hospitals. In any event, the

76 SAVILLE KUSHNER

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

generalisation (telling nurses and doctors in another hospital that thesecompetencies ‘work’) would gain limited purchase on a new contextsince the political/professional cultures will be different and the nuancesin the competency framework may not match the second hospital cul-tures. The epistemological assumptions of the evaluation method do notmatch the forms of practical knowledge at play in the context.

2. We may assume that these learning/practice interactions in respect ofcompetencies are more complex than can be accounted for by linearcausality or first-order calculations of ‘what works’. But the enquiry, asit broadens out to explore unanticipated variables (like politics, gender,professional aspiration) and starts to publish initial thoughts about howcompetencies are contingent upon professional and institutional rela-tionships, attracts the active interest of nurses, doctors and administra-tors. People become less sure about what works but better informedabout how nursing competencies work � the impact of competencytraining becomes uncertain and ambiguous as people come to reflect cri-tically on purposes and significances and multiple ways of knowing com-pete for dominance in the task of asserting explanation.

It is the second option that case study theorists went for. But they did notabandon the possibilities of generalisation � far from it. In fact, by usingcase study approaches they expanded the range of valid possibilities for gen-eralisation. Once having documented contingent relationships in enoughdetail to understand the interplay of many variables, and having come tounderstand observable action as an effect, as it were, of its contexts, theycould tell a story. That story, since it reflected the subtle and cultural knowl-edge of practitioners, would be compelling to the actors since it illuminated,affirmed and revealed preoccupations they had, judgements they madeand what they were trying to do. As MacDonald (1977) wrote, the aim wasto ‘shift the locus of responsibility for generalisation to the practitioner’.

Now, the following forms of generalisation become available:

1. Within the case � Campbell’s ‘corroborated locally’:• A nurse can generalise his dilemmas and aspirations in terms of an

institutional discourse � that is, he can locate himself in the currentof influential thinking and power. The case study narrative will maketransparent links, for example, between policy, organisation and prac-tice (shifting ‘up’ and ‘down’ the system, e.g. is a form ofgeneralisation);

• Where the case study narrative is shared, it helps people shapetheir views and professional commitments through more informed

77Case Study as Antidote to the Literal

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

interaction with each other and in ways more likely to be effective ina known professional culture. This empowers not only the individualbut also the organisation. Horizontal movement of ideas is also aform of generalisation;

2. From one case to another � Stake’s ‘naturalistic generalisation’• Competency trainers in another hospital might well conclude from

reading the narrative, ‘yes, I see the point � what lies behind competen-cies is just that sort of professional politics. Useful, that…’;

• Those trainers might say, ‘quite different here � the case study just isnot relevant � but what an interesting way of analysing our workingcontexts � we could use that’;

• They might say, ‘no � they have it wrong � now we can see clearlyhow not to go about it � they are using competencies in the wrong way.Mind you, we’re also developing a medical training program and thatstory might be more useful with those guys � I’ll pass it on’;

• Or two trainers talking at a conference might share their thoughts: ‘ah� too difficult for us � we’re going down a different route. Didn’t likethe way they treated nurses’. ‘Really? We were quite impressed, actually� nurses felt it made things more concrete � we’re planning to learnfrom that’. ‘Oh? Tell me how’.

These are all forms of contingent generalisation � much reflectingStake’s ‘naturalistic generalisation’ (i.e. generalisation through receptionand recognition rather than through demonstration and dissemination,Stake, 2004). Other professionals might find ways of usefully transfer-ring an illumination of ‘the way they treat nurses’ to their own con-texts. But it is important to notice that the form taken by generalisationin each of these instances is one of situational theorising, continuity ofanalysis from the case research to its reading, each time based on criti-cal analysis of contingent relationships. The case study story provokesreflection on action and circumstance, and as people come to a decisionon the utility of the case study (or otherwise), those reflections crystal-lise into a theory of organisation and action. MacDonald (1977) put itlike this:

If … we shift the burden of responsibility for generalizing from the outsider to the insi-

der, from the evaluator to the practitioner, and if we restrict the task to that of general-

izing from one fully described case to another that is fully known [e.g. the one in which

these competency trainers live] then we can argue that portrayal of a single case may

still fulfill the function of generalization, though it calls for a redistribution of responsi-

bilities with respect to the evaluation process.

78 SAVILLE KUSHNER

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

This is an extension of Campbell. It is precisely the local corroborationof valid insight that provides the basis for generalisation to other contexts,so long as by generalisation we mean ‘illumination’ and not ‘reproduction’;so long as we are transferring from one context to another both the insightand its contingent ephemerality. What makes transfer possible is that theanalysis of contingency taps into and, at best, replicates, both the way prac-titioners think through their options, and the conditions within which thoseoptions vary. One classroom may be a reality apart from the one nextdoor � but similar to one on the other side of the world. The curriculumethic may differ � but there will be an ethic; pedagogical authority will dif-fer, but authority will be a determining factor in learning; life experienceswill vary enormously, but all classrooms will raise issues about theirrelationship to life. But more revealingly, that initial classroom may bemore like a ward in the nearby hospital than it resembles the adjoiningclassroom if dominant realities concern a relationship of rights and dutiesbetween practitioners and patients/pupils or the tension between the practi-tioner’s ethic and that forced on them by coercive authority or competitionbetween different ways of knowing between practitioner and patient/pupil.

It is the combination of the explanatory power of contingency analysisand its verification through interaction and collaborative analysis thatallows for generalisation from cases. The restrictions of literalism � theattenuation of variables, the freezing of context, the hierarchy that givespriority to pre-specified questions and relegates situational insight � act asa constraint to such generalisation. There are situations where sufficientvariables can be ‘controlled’ to allow for the specification of an effect, butthese cannot include local studies of human interactive practices.‘Controlled, feature-at-a-time comparisons are vulnerable to repeated decom-position: there are features within features, contingencies within contingen-cies, and tasks within tasks’ (Greenhalgh & Russell, 2010). Or, moreprecisely, in these local situations we can invest in identifying and control-ling variables, but we are unlikely to make generalisations whose validitywithstands the judgement of those who practice and live in the situationsdescribed � because it is not just the variable itself that is to be neutralised,it is the interaction and feedback between variables. To repeat Cronbach(above), ‘A general statement can be highly accurate only if it specifies inter-active effects that it takes a large amount of data to pin down’.

It is precisely in this sense that we can see the power in Campbell’sanalysis that, ‘The theories and hunches used by those who put the thera-peutic package together must be regarded as corroborated, however tenta-tively, if there is an effect of local … validity’. Where we may elaborate

79Case Study as Antidote to the Literal

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

his thinking is to shift his suggestion that ‘we must back up from the cur-rent overemphasis on theory’ to one that says, ‘we must move from ‘the-ory’ to ‘theorising’ � that is, the practitioner as interpretive agent oftheir own context. Here is the real power of evaluation case study and itsfocus on contingency. It is precisely this possibility � building a commen-tary on interactions � that is lost in literalism. In fact, we might go sofar as to say that literalism displaces theorising through its narrow cap-ture of limited variables and with its linear (cause-effect; independent-dependent variation) rationality.

DUALISM AND THE PSYCHOLOGICAL

COMMITMENT

All dualisms break down empirically. Literalism/contingency no less,though I have insisted that these are options, not dualisms � dualities, per-haps, allowing us to focus on the dynamic tension between the two. Westruggle to make sense of both, to be fair to the question and its alterna-tives, to balance what people want with what they need, noticing the biggerpicture but not having the confidence to smuggle it into the report. Here isanother way to think of case study � as a site in which these competingethics are at play, in which these tensions are resolved. As Stake says ofsuch a dualism, we need ‘binocular vision’ � to use stereoscopic methods toachieve depth of field. Not to put a distinction on action and context (fore-ground/background) is subtle, requires an eye for nuance � and the time toexercise it. It, too, is an attractive dualism that also breaks down underscrutiny.

In the bilingual case, we re-entered the school and redesigned the evalua-tion study guided by the teachers’ insistence that they be represented intheir own terms � that is, meaningfully. But the design was stillnegotiated � it was a mutual education project. We observed informalinteractions, talked with teachers about themselves, their lives and beliefsand used these to reflect on their professional practices. Personal portrayalsof teachers emerged as we moved, as we wrote, ‘from curriculum vitae tocurriculum’ to understand the sources of pedagogical action in personal andcollective experience. The literalism of our initial sociolinguistic approachfell away in significance as we engaged in negotiated accounts of their prac-tices as defined more by values than by techniques. We emerged with atheory of bilingual pedagogy unique to that school, but available to others

80 SAVILLE KUSHNER

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

who might find that they could usefully generalise the theory to their owncontext. What allowed the possibility of such ‘naturalistic generalisation’was that we documented action, not in the context of theory, but in thecontext of day-to-day realities � the practical challenge of professionalaction. We left the sociolinguistic observations in the case study, but as aprovocation to the reader to consider in a comparative sense how bilingualeducation (and other aspects of civil rights education) resolves the tensionbetween what practitioners are required to do (to act in literal compliance)and what they do as a preference (engage directly with contingentrelations � such as building strong, often maternal relationships with kidsas a basis for developing confidence in taking on language challenges). Thereality of the practices we observe in evaluation case study is frequently likethis � balancing requirement with preference, compliance with creativity,literalist definitions of professional action with contingent (situational)definitions. Case study as an emergent design methodology allows forcasting method in the image of action.

We are victim to mind-sets. We have emotional and ideological attach-ments, we are steeped with greater or lesser courage, we are fortunate tohave curiosity but plagued by the limits of our imagination. It is whenwe enter the field that we commit emotionally and psychologically to apoint of view. Do we have the tolerance for uncertainty that attends casestudy? Do we feel ethically bound to work within the limits of a literalistcontract? Do we feel that we have the imagination to enter into uncer-tainty and represent it in narrative form? Preparing for entry to the fieldis when we ‘set’ our ‘minds’. Contracted evaluators’ conversations withsponsors, or students’ with their supervisors alert us to the room formanoeuvre, the limits of political and institutional tolerance to tarry longenough to consider those nuances and to measure the forbearance foruncertainty. We negotiate with our team colleagues and with ourselvesour levels of courage and commitment to push at an edge, to risk reputa-tion, to take additional time to learn and to shape our purposes ratherthan take them as given. I have too infrequently taken up the challengein recent years, respectful sometimes of colleagues’ caution or of my ownconservatism or of the fragile political standing of my professionalsponsors � I have too frequently withdrawn � always consciously. Ineach and every case, the evaluation has suffered limitations as a result,its reach of understanding noticeably attenuated, too many of its findingswholesome but work-a-day, important themes sensed but suppressedthrough lack of investigatory perseverance. I have found taking the eva-luation question too literally has sacrificed analytical quality, that good

81Case Study as Antidote to the Literal

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

understanding has been limited by colleague’s unwillingness to collectpersonalised data, while not yet knowing what its significance might be.

One lesson I have learned is that case study is made immeasurably easierin a working culture where there is a collective will to explore the ideas andthe practices. In fact, I would go further to say that evaluation case study isonly feasible as a sustained practice where there is a coincidence of theoris-ing, practising and teaching about evaluation. Each of those activities pro-vides a site for contemplation and experimentation, argument, testing eachother, action-with-reflection and curiosity about failure. Teaching givesthe opportunity to articulate and adumbrate on your experience, explore itand get feedback. Theory and practice represent another dualism, equallybrittle, broken and reintegrated through teaching.

REFERENCES

Besecke, K. (2007). Beyond literalism: Reflexive spirituality and religious meaning. In

N. T. Ammerman (Ed.), Everyday religion: Observing modern religious lives. Oxford:

Oxford University Press

Campbell, D. T. (1999). Relabelling internal and external validity. In D. T. Campbell & M. J.

Russo (Eds.), Social experimentation (pp. 111�122). Thousand Oaks, CA: Sage.

Cronbach, L. J. (1975). Beyond the two disciplines of scientific psychology. American

Psychologist, 30(2), 116�127.

Dougherty, V. M. (1994). Absurdity and the limits of literalism: defining the absurd result

principle in statutory interpretation. American University Law Review, 44, 127�166.

Greenhalgh, T., & Russell, J. (2010). Why do evaluations of eHealth programs fail? An alter-

native set of guiding principles. PLoS Medicine, 7(11), e1000360.

Jocher, K. (1928). The case method in social research. Social Forces, 7(2), 203�211.

Kushner, S. I. (2000). Personalising evaluation. London: Sage.

Linder, M. (2005). Nothing less than literal: Architecture after minimalism. Cambridge, MA:

MIT Press.

MacDonald, B. (1977). The portrayal of persons as evaluation data. In N. Norris (Ed.),

SAFARI two: Theory in practice, CARE Occasional Publications #4. Norwich:

University of East Anglia.

MacDonald, B. (1987). Evaluation and the control of education. In R. Murphy & H. Torrance

(Eds.), Issues and methods in evaluation (pp. 36�48). London: Paul Chapman.

MacDonald, B., & Kushner, S. I. (1982). Bread and dreams: A case study of bilingual schooling

in the USA, CARE Occasional Publications #12. Norwich: University of East Anglia.

Parlett, M. R., & Hamilton, D. (1972). Evaluation as illumination: A new approach to the study

of innovatory programs. Edinburgh: University of Edinburgh, Centre for Research in

the Educational Sciences.

Russell, J., Greenhalgh, T., Boynton, P., & Rigby, M. (2004). Soft networks for bridging the

gap between research and practice: Illuminative evaluation of CHAIN. British Medical

Journal, 328, 1174–1177.

82 SAVILLE KUSHNER

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)

Shaw, C. R. (1927). Case study method. Publications of the American Sociological Society,

XXI, I49�158.

Stake, R. E. (2004). Standards based and responsive evaluation. Thousand Oaks, CA: Sage.

Stake, R. E. (2010). Qualitative research: Studying how things work. New York, NY: Guilford

Press.

Wolcott, H. (1973). The man in the principal’s office: An ethnography. New York, NY: Holt,

Rinehart, & Winston.

83Case Study as Antidote to the Literal

Dow

nloa

ded

by P

rofe

ssor

Sav

ille

Kus

hner

At 1

3:21

20

Janu

ary

2015

(PT

)