Naturalistic inquiry applied to the evaluation of a teacher education program

NATURALISTIC INQUIRY APPLIED TO THE EVALUATION OF A TEACHER EDUCATION PROGRAM

DONNA L. AKSAMIT

University of Nebraska-Lincoln. U.S.A

S. PIKE HALL

Drake University, Iowa, U.S.A

LYNNE RYAN

Providence College, Rhode Island, U.S.A

Abstract - A team of six teacher educators (in addition to the authors. other team members are Susan Peters, Michigan State University; Ginger Fisher, Holy Family College, Philadelphia; and Tom Skrtic, University of Kansas) was invited to evaluate an innovative teacher education program which has merged elementary and special education into one major. Naturalistic inquiry methodology was employed to evaluate the program. This paper (a) presents a theoretical and philosophical discussion of naturalistic inquiry and differentiates it from quantitative methodology. (b) takes the reader through the entire naturalistic evaluation process from the initial site visit through the writing of the case study and the audit, (c) provides a brief description of the organization of the case study and a limited discussion of the major findings, and (d) discusses implications of naturalistic inquiry as they apply to this particular study and to evaluation of teacher education programs in general.

The major focus of this paper is on the use of naturalistic inquiry to evaluate a teacher education program. Although qualitative research has become increasingly popular for investigating education issues, including those in teacher education, the majority of teacher education research continues to be of a conventional or quantitative nature. A review of the major teacher education journals indicates that com- paratively few faculty in institutions which prepare teachers routinely use qualitative research. Additionally, the inclusion of qualitative research courses in doctoral programs which prepare future teacher educators appears to be occurring at a slow pace.

Fetterman (1988) has pointed out that qualitative research is not a monolithic entity. A multitude of qualitative approaches exist (e.g., ethnography, naturalistic inquiry, connoisseur- ship, and field research) and are used in a variety of ways in education. The qualitative ap-

preach used to evaluate the teacher education program investigated in this study is best described as “naturalistic inquiry” (Guba, 1981, 1987; Guba & Lincoln, 1981, 1988; Lincoln & Guba, 1985; Skrtic, 1985). Naturalistic inquiry distinguishes itself from other qualitative methodologies in its tight adherence to a constructivist set of assumptions and a set of quality criteria which demand great rigor as the approach is applied to program evaluation.

The teacher education program evaluated is a combined elementary/special education major. Preservice teachers are prepared to teach both regular and special education students through a truly “merged” program. The program exists in the teacher education department of a medium-sized (3200) liberal arts college located in a large eastern city in the U.S.A. Numbers of graduates ranged from 12 in 1987 (the first group to complete the merged major) to 35 in 1989. The evaluation was done by six teacher

215

educators from six programs of varying sizes, structures, and geographical locations who were interested in investigating alternative models of preservice teacher education, particularly those designed to prepare individuals to teach at-risk and other special needs children.

The paper has four sections. Section one presents a theoretical and philosophical discussion of naturalistic inquiry and compares it to and contrasts it with the more conventional and positivistic quantitative methodologies. The adherence to the constructivist set of assumptions and the parallel and unique criteria of naturalistic inquiry are also addressed.

The second section of the paper takes the reader step by step through the naturalistic process. The philosophical and theoretical discussion in section one is connected to and reflected in this application section through references to naturalistic evaluation’s criteria of quality as they were applied to the evaluation of this particular teacher education program.

Section three provides a brief description of the organization and content of the case study which resulted from the evaluation as well as a limited discussion of the major findings. These findings focus on the four major themes identified as central to the merged program. The case study, with complete results and methodology appendix, is being published elsewhere in the form of a monograph. *

Section four discusses implications of naturalistic inquiry as they apply to this particular study and for use in evaluation of teacher education in general.

Theoretical and Philosophical Context

Those of us who advocate understanding of “things” as they relate to each other and as they relate to the wholes of which they are a part have often been frustrated by research formula- tions which are designed for minute inspection of parts. Such an endeavor might produce a clear yet “untrue” image. Clearly what is “truth” is a matter for considerable debate, but what we are trying to make clear is that there is a perspective on truth which does not accept

216 DONNA L. AKSAMIT et al.

that a view of a part in isolation from its embrac- ing whole is true, that is, that view is unlikely to be useful or supportive of productive interaction with that whole or any of its other parts.

For example, if an evaluation team were to visit college X’s teacher education program and scrutinize its faculty’s curriculum vitae as the necessary and sufficient measure of the worth of the program, most of us would object. Surely, the objectors would argue, there is more to worth than whether Y% of the faculty have received their doctorate. The objectors would further state that the uselessness, the medioc- rity or the greatness of a program must be viewed as an interactive product of many variables. Among them, for example, might be re- sistance to or support of the program by central administration, the entry level achievements of the prospective teachers, the relevance and effectiveness of the faculty’s instructional approaches and the “culture” of the teacher education program. In short, when considering human systems, we would not generally argue that a situation can move forward based on interventions with only one of the complex of variables which affect the production of desired results. In most human systems there are too many variables for simple understandings or simple interventions to be the rule.

The philosophy of naturalistic evaluation and research has been constructed in recognition of these and numerous other understandings which are not easily reconciled with linear, quantitatively and/or probabilistically oriented cause-effect schemes as applied to human actions or organizations. The distinctiveness of the naturalistic approach will become apparent when the naturalistic and conventional approaches to understanding and to evaluation and research are compared.

Contrasts of Ontological, Epistemological, and Methodological Premises of Conventional and Naturalistic Inquiry

The conventional (also referred to as “positivistic” and “rationalistic”) and naturahs- tic views are so distinct as to be, for most pur- poses, opposed (Guba, 1987). This becomes more obvious when the ontological (what can

*Contact Dr. Donna Aksamit, 202 Barkley, University of Nebraska, Lincoln, NE 6858341732, U.S.A

Naturalistic Inquiry 217

be known), epistemological (the relation of the knower to the known), and methodological (how the inquirer can acquire knowledge) post- ulates of each are examined (see Table 1).

Naturalists argue that while there are objects, events, and processes with which people in- teract, it is the meanings that people ascribe to events or the interpretations that people make of processes that are of interest in human events (Guba, 1987, pp. 81-82).

Thus we come to the premises that guide the work of the naturalistic evaluators who use qualitative methods to aid their “thinking naturalistically.” For such persons the naturalistic members of the axiom pairs in Table 2 make

Table 1

The Conventional and Naturalist Belief Svstems (Cuba. 1987)

considerable sense. In short, the evaluator will work first to bring

to the surface constructions of the ‘various stakeholder groups, then explicate those constructions collaboratively with the stakeholding groups and, finally, support negotiated communication between constructor groups until they are able to construct a single, consensual, and local reality which all will accept.

As with conventional evaluation, the naturalistic evaluation process may be conducted poorly or well, with results that are desirable or not. The criteria for judging quality in naturalistic and conventional endeavors are compared in the next section.

Conventional posture Naturalistic posture

Ontology Realist: Relativist:

There exists a single reality independent of any There exists multiple social constructed realities observer’s interest in it that operates according to ungoverned by any natural laws immutable natural laws many of which are causal in form

Truth is defined as facts isomorphic with reality Truth is defined as the most informed and sophisticated constructions on which there is consensus among qualified critics

Epistemology Dualist, Objective: Monist, Subjective:

It is possible for an observer to exteriorize the reality The inquirer and the inquired-into are interlocked in studied, remaining detached from it and involved with it such a way that the findings of an investigation are the

literal creation of the inquiry process. (This assertion obliterates the ontology/epistemology distinction)

Methodology Interventionist: Hermeneutic:

The context is stripped of contaminating (confounding) The context is construed as giving meaning and influences so that the inquiry can converge on truths, existence to the inquired-into; the methodology explaining nature as it really is, leading to the involves a dialectic of iteration, analysis, critique, capability to predict and control reiteration reanalysis, and so on, leading to the

emergence of a joint (combined emic/etic) understanding of a case

Table 2

The Conventional and Naturalistic Evaluation Axioms (Freely Selected from Cuba, 1987, p.37)

Conventional posture

Evaluators can find a place to stand that will provide the leverage needed for the objective pursuit of evaluation activities

Naturalistic posture

Evaluators are subjective partners with stakeholders in the literal creation of evaluation data

Evaluators are the communication channels through which literally true data are passed to the audiences of the evaluation reports

Evaluators are orchestraters of a negotiation process that attempts to culminate consensus on better informed and more sophisticated constructions

218 DONNA L. AKSAMIT et al

Naturalistic Evaluation’s “Criteria of Quality”

Advocates of naturalistic evaluation are in initial phases of their development of a unique and appropriate set of quality criteria. Initially, proponents developed criteria of trustworthiness of adapting the general notions behind the conventional qualities of internal and external validity, reliability, and objectivity. Sub- sequently, Lincoln and Guba (1986) proposed criteria of authenticity which are not only unique to naturalistic evaluation but, more impor- tantly, are in a more exact proximity of the central propositions of naturalistic evaluation. Both the parallel and unique criteria of quality applied to naturalistic reseach are discussed below.

Parallel criteria of rigor. The naturalistic parallels to conventional or rationalistic trustworthiness are credibility (internal validity), transferability (generalizability), dependability (reliability), and confirmability (objectivity).

The credibility component of trustworthiness has been defined as a “check on the isomorph- ism between the inquirer’s data and interpretations and the multiple realities in the minds of informants” (Guba, 1981, p. 84). In general, an evaluator seeks constantly to note connections of conclusions with data. Further, the evaluator seeks prolonged engagement with the natural setting as a means of assuring that the generali- zations being produced are at least like, if not equivalent to, those that a stakeholder in the setting would make. A continual checking with respondents takes place in order that the naturalistic observer may assure that appropriate constructions are being recorded and abstracted from the situation (appropriate being locally relevant and “in tune” with the thinking of the stakeholders).

Transferability has been defined as the “equivalent of generalizability to the extent that there are similarities between sending and receiving contexts” (Guba, 1988, p. 84). Here Guba proposes that the primary assurances of the de- gree of transferability of the findings be (a) purposive/theoretical sampling for maximum variety of opinion within and between stakeholding groups; and (b) thick description, that is, providing the reader of the case report with suffi-

cient information to vicariously experience the reality of the evaluand.

Dependability “includes the instability factors typically indicated by the term ‘unreliabil- ity’ but makes allowances for emergent designs, and developing theory that also induce change but which cannot be taken as ‘error”’ (Guba, 1988, p. 84). Lincoln and Guba (1986) have proposed two processes as assurances of dependability. First, stepwise replication, in which evaluation and data sources are split in half and, providing that there is sufficient inter-team communication to allow articulated development of the emergent design, pursue the inquiry independently. Second, the dependability audit is used. In this the study’s auditor follows the team’s carefully laid “paper trail” (in which all of its data, decisions, and emergent decisions are documented) to verify that the best accepted practices of naturalistic evaluation have been followed.

Confirmability “shifts the emphasis from the certifiability of the inquirer to the confirmability of the data” (Guba, 1988, p. 84). For this purpose, an auditor is employed to verify that categories, supracategories, inferences, and conclusions have been produced through plausible manipulations of real data. He or she does this by means of the paper trail described above. Here, however, the concern is less with pre- scribed practices than unbiased manipulating of data. In examining the paper trail the auditor will be able to ascertain whether or not bias has entered the picture.

Authenticity. The criteria of quality which are unique to naturalistic inquiry are fairness, ontological authenticity, educative authenticity, catalytic authenticity, and tactical authenticity.

Fairness is concerned with treating data, constructions, and the negotiation concerning constructions, in such a way that no one individual or stakeholding group is empowered over others and that no individual is impoverished through the process. There can be no guarantee that this will result, but every reasonable effort must be made to “avoid, at least probabilisti- tally, the possibility that certain values will be diminished (and their holders exploited) while others will be enhanced (and their holders ad- vantaged)” (Lincoln & Guba, 1986, p. 79). Cer- tain process criteria must be met for the ba-


lanced presentation of the stakeholder’s constructions and for the management of the negotiation process that is intended to lead to consensus.

The evaluator hopes, by means of the data gathering, explication and negotiation processes to give the persons involved in the study a broadened experience and interaction with others as well as an experiencing of an enhanced understanding of the consequences of various personal actions and beliefs. The criterion of ontological authenticity is satisfied if the evaluation produces an improvement in the sophistication and comprehensiveness of the participating individual’s consciousness.

The criterion of educative authenticity implies that there will be an increased understanding not only by the individual of herself or himself but of the constructions, together with their rootedness in the values, of other stakeholder groups. This does not imply a liking for such constructions but does imply an enhanced sophistication and understanding within a “story-line”, because of the increase in understanding. How this standard is to be met is as yet not clear (Lincoln & Guba, 1986, p. 82) even for small groups. Some provision must be made for the findings to be presented, perhaps to a small group of influential representatives of the various stakeholder groups, in such a way for them to be educative in the sense described here.

Catalytic authenticity is met if the evaluation not only educates, but also stimulates actions. While methods for assuring this result are as yet underdeveloped, it is assumed that the methods devoted to meeting both parallel criteria and other authenticity criteria will go some distance in assuring that this standard is met.

While prompt action on the part of stakeholder groups would be enough to satisfy the criterion for catalytic authenticity, that action would not, alone, satisfy the criterion of tactical authenticity. That action should be action that results in a desirable change, enhancing the em- powerment of one or more stakeholding groups. For example, naturalistic evaluations are enjoined to move in this direction (Lincoln & Guba, 1986, p. 83) by giving each stakeholder an opportunity to control the evaluation. Ini- tially this is done by having each stakeholder own the data that derives from the interview. This ownership is supported in the explication

and negotiation process (member check) which occurs later in the naturalistic process.

Clearly, then, the naturalistic evaluator/ evaluation team expects to have a relationship with an evaluand that is far different from the evaluator/subject relationship of the conventional evaluation. A naturalistic evaluator is equipped with a variegated set of meta-values which span the distance between concern for ef- ficiency and effectiveness in evaluation, on the one hand, and for evaluation and research which improves the consciousness and the sociopolitical welfare of the stakeholders, on the other.

Naturalistic Evaluation of the Merged Program

Our entry into and progression with the exp- loration of a merged elementary/special education program would not be standard. We could not provide a set script for our actions, nor could we promise or predict or limit ourselves to the sort of findings that would result from our labors. We were at the site as much as learners as teachers; as storytellers and synthesizers. In- itially we would enter as learners, asking that the various stakeholders teach us what must be known about their program.

Preparation for the First Site Visit

Prior to the first site visit documents, such as college catalogs, minutes of departmental meetings, course descriptions and syllabi, faculty vitae, published articles by faculty and jour- nalists about the program, department self- evaluations, state accreditation reports, and a student teacher handbook, were reviewed, analyzed, and discussed by team members. From this view, decisions about who would be interviewed and observed were made purposefully, with representatives from all groups of stakeholders (e.g., faculty, cooperating teachers, students, school administrators, and graduates for the program) included. Questions and issues were developed and schedules for interviews and observations were made.

Site Visit One

During the first site visit, which was held in


April 1988, all faculty involved in designing and teaching in the merged program were interviewed and selected faculty were observed while teaching program classes. Students cur- rently enrolled in the program and those who had graduated and were teaching in schools within the geographic area were also interviewed and observed. In addition, university and state department personnel, school administrators, and cooperating teachers were interviewed.

In order to ensure that the issues that emerged from the interview were those of the stakeholders, interviewers began the interview with general statements such as “If I was telling the story of the merged elementary/special education program, what do you think people would want to know?” More specific questions, generated from the documents and from previous interviews, were asked only after the respondent had an opportunity to respond to the more open-ended questions.

Each team member kept a field journal of observations, interviews, and document reviews. As is consistent with naturalistic inquiry, data analyses occurred continuously. The inquirer’s comments and reactions, as well as notes on additional individuals to interview (to get similar and alternative points of view) and additional documents to review were bracketed within the field notes. By cross-checking data from multiple sources, the team addressed a critical component of naturalistic inquiry “credibility,” that of triangulation.

After each day’s activities, team members shared field notes and reactions and discussed and debated findings and issues for documentation of trustworthiness, credibility, and dependability. These discussions (as were all team discussions) were taped and later transcribed and provide “a log of developing insights and hypotheses” (Lincoln & Guba, 1982, p. 11) which will be useful in the post hoc auditing pro- cedures. In addition to field notes and record- ings of discussions, logs ivere also kept of day- to-day activities of each team member during field visits, of activities related to the study after the field visits, and of methodological decisions.

At the end of the first site visit the first “reality check” was conducted in the form of an exit debriefing in which team members shared the story as it had unfolded at this point. A note of

caution was sounded that there were still many gaps in the story and that a more thorough review of the data, which would be occurring in subsequent weeks, would reveal other issues and/or modify the team’s perception. This debriefing not only gave the researchers an opportunity to begin to provide the respondents with a mirror of their realities and values, but it also enabled the researchers to present the multiple perspectives of the various stakeholders and to help them interpret these perspectives, thus contributing to educative authenticity. At the completion of the exit debriefing preliminary arrangements and timelines were set for the second site visit.

Between Site Visits

Unitizing, coding, and individual categorizing of data. Following the initial site visit, each team member formulated field notes into discrete units of information which were placed on com- puter cards (units = smallest unit narrative which is self-explanatory). In addition to the narrative, each card unit also contained iden- tifying information such as the interviewer, site, site visit number, respondent initials, role (e.g., teacher, student, etc.), community, school, and source (e.g., interview observation, document analysis),

All document and field note units were also coded and these codes were listed on the card units so that data and findings could be traced during the audit check. These coded units were then categorized by the individual team member into groups of similar content.

All cards were filed in a computerized data base. Pertinent data, such as respondent or document source, interviewer, date, et cetera, as well as multiple cross referencing of the card (and its narrative) to the team’s emerging category system were all entered on each card. The system allowed the projection of each card and/ or summaries of sets of cards on a wall screen for team viewing at the team categorization meeting.

Team categorization. The team met after all cards had been entered in the data base and were ready for review. From this review categories were combined and new ones were added


to make a better fit of the cards within a set or spectives of the stakeholders and thus contri-

category. buted to the credibility of the story.

Isolation of themes and questions. From an analysis of the projected units and preliminary clusters, members identified eight major categories or themes. Questions relating to gaps and inconsistencies in each of the eight categories were formulated to identify missing data or in- complete information about relevant issues. Also, additional individuals or role groups to be interviewed and/or observed and further documents to be reviewed were identified.

Draft protocolslinterviews. Prior to the second site visit, individual team members developed draft interview and observation protocols. These protocols were reviewed by all team members and were used as a guide during interviews to acquire the additional information on the themes which had emerged during the first site visit.

During the second site visit team members at- tempted to determine how, or if, the analyses shared during the first site visit, either through the exit debriefing or the didactic interactions during the interviews of individual stakeholders, affected the program. A number of examples, such as the development of a revised student teacher competency checklist and the initiation of discussions by the teaching education faculty on the theory or theories, driving the merged program, were found as evidence that the evaluative process met Lincoln’s (1986) catalytic authenticity criterion (requiring that the evaluation itself must stimulate action).

Post Site Visit Two Activities

Second Site Visit

The strategy during the second site visit was to move from the open-ended approach noted above (i.e., “to tell the story... .“) to a somewhat more structured second round of data col- lection in order to focus on the major themes and to probe for increasingly specific and detailed responses which provided richer data and a more complete “picture” of the program.

After the second site visit members again unitized and categorized field notes. However, this time individual members had the outline generated after the first site visit to guide categorization efforts. A team meeting was held in January of 1989 to review data and revise Site Visit Two categories. Team members then col- lectively reviewed, discussed, and analyzed all unit cards and categories. At this point the orig- inal eight categories/themes were collapsed into four program themes (these are discussed in section three). From this exercise a case study outline was developed.

For example, one of the major topics identified when the first site visit data were unitized and categorized was the “developmental approach” which, according to the stakeholders, is the foundation of the merged elementary/special education program. Instead of the more open-ended questions of the first site visit, several interview questions during the second site visit were more specifically focused on the developmental approach. Questions such as “How do you define the developmental approach as it applies to your merged program?” or “What skills or competencies does a teacher trained in the developmental approach de- monstrate in the classroom?” were included in the developmental interview protocol. Re- sponses to these and similar questions would reveal critical features of the “developmental approach” as viewed from the multiple per-

The draft case study. The next step in the process was the writing of a draft case study based on the outline developed following the second site visit. Because team members are geog- raphically separated, sections of the draft were completed independently, each section was reviewed and revised by other team members, and the final draft was then completed.

It is at this point, of writing the case study, that tacit knowledge and values of the inves- tigator(s) come to bear. While the writer’s understanding and translation of that understanding is to be grounded in the data, there is no attempt to separate that tacit knowledge and those values. There must be careful presentation of data through direct quotation and other special methods (e.g., specific reference to the interview, observation, and/or document review from which the data were derived) thus

222 DONNA L. AKSAMIT et al.

making clear the grounding of the overall expla- nation, description, and assertions set forth in the case study (Skrtic, 1985). This careful documentation of findings coupled with the re- porting and checking of findings based on the multiple perspectives (i.e., triangulation) con- tributes to the perception of trustworthiness, or to use rationalistic terminology, the internal validity, of the study.

Third Site Visit

Member check. The draft case study was sent to all stakeholders 2 weeks prior to the member check visit. Representatives from all stakeholder groups, including those who had been interviewed and/or observed as well as a sample of those who had not, were invited. A total of 24 faculty, students, graduates, cooperating teachers, and building administrators attended the member check meeting. In addition to the draft case study, all participants received in advance: (a) a General Comments Form on which they commented on the overall credibility of the draft case study, and (b) a Response to Qual- ifiers Form which included 50 statements that were found in the case study and which expres- sed beliefs or positions held by some stakeholders as derived from interviews. Stakeholders attending the member check session made general comments and also commented as to whether or not they agreed with the specific statements and also commented as to whether or not they agreed with the specific statements on the Qualifier Form. They then made corrections which reflected their opinions/positions for the entire case study. These forms and pro- cedures assured all stakeholders an opportunity to express their views in writing as well as during the actual member check discussion.

The member check meeting consisted of three major phases. First, each stakeholder was asked to comment on the overall correctness of the report. The option to pass was respected. Second, they were asked to raise any specific issues or concerns about the total case study. Third, the qualifier statements were reviewed, with all stakeholders using the forms they had completed in advance as a guide. The role of the naturalistic inquiry team throughout this member check process was to listen to the stakeholders’ comments and corrections, probe

for additional understanding when needed, and summarize and clarify stakeholders’ viewpoints. All proceedings were taped and transcribed. In addition, team members took field notes as a backup for the recording.

Copies of the draft case study which had been distributed, as well as both the General Com- ments and Qualifier forms were collected at the conclusion of the member check session. Both verbal (as taken from transcribed tapes and field notes) and written comments and corrections (taken from the General Comments and Qualifier forms) were categorized and sum- marized by team members. Follow-up phone interviews clarified verbal and written comments as needed. The case study was then revised based on comments and corrections and subsequent discussions. Where there were differences in perceptions and conclusions between the evaluation team and a stakeholder( multiple viewpoints were included in the final case study.

The member check is a critical component of naturalistic inquiry and one of the major factors that distinguishes this qualitative process from others. The member check described above is actually one of a series that occurs continually throughout the process (Skrtic, 1985). Others include: (a) each didactic interview when interviewers ask stakeholders to verify their own data and to confirm or deny data collected from interview with other stakeholders; and (b) the exit debriefing at the end of the first site visit. The member check process thus allowed data to be continually checked and contributed to the credibility (internal validity) of the study. Through the member check processes, members were also brought to an increased understanding of their own (ontological authenticity) and others’ (educative authenticity) points of view. Assuring the accuracy of the representa- tions of stakeholders’ constructions addresses the authenticity criteria of fairness. To mis- represent stakeholders’ points of view, either inadvertently of purposefully, decreases their access to “power.”

The Audit

A defining step of each naturalistic study is the audit. An audit is designed to judge the rigor (parellel criteria) and authenticity (unique criteria) of the study. The raw material for this


audit is the research team’s paper trail which documents data (raw field notes of observations and interviews and documents), decisions, grounded theories, and emergent designs (coded units, categories, transcriptions of team meetings, team correspondences, logs and journals of individual members, and the case study). Through examination of this audit trail, the auditor (similar in concept to the fiscal auditor) as- sessed the credibility (persistent observation, triangulation, and member check), transferability (purposive sampling and thick design), dependability (have the best accepted practices of naturalistic design been used in the study?), and confirmability (have the categories, supracategories, inferences, and conclusions been produced through plausible manipulations of the data?) of this study using the standards and process developed by Halpern (cited in Lincoln & Guba, 198.5). The audit for the study was done by an independent professional trained in naturalistic inquiry but not previously involved with this particular program evaluation. Fol- lowing the audit, the case study was dissemi- nated to all stakeholders and published as a monograph.

perspectives of the multiple stakeholders identified previously. It is in this section that varia- tions in perceptions, understandings, and opinions among stakeholders, when and if per- ceived, were presented. Section Four discussed Program Impact by asking and then responding to two main questions: (1) Is the ESE program producing developmentally oriented teachers? (since the developmental approach is the foundation of the program), and (2) Are the developmentally oriented teachers (if they exist) being “swallowed up” by the traditional school system? Section Five was titled Lessons Learned, and provided four major recommendations for future program directions.

Findings: Four Major Program Themes

The four major themes which emerged as stakeholders told their story were: (a) the developmental orientation underlying the entire ESE program, (b) the focus on preparing teachers to be critically reflective, (c) the grounding of the program in effective teaching research and literature, and (d) the distillation of program understandings through field applications at many levels.

The Case Study Developmental Approach

Organization and Content

The draft case study consisted of five major sections. Section One, Introduction, included such topics as site description, student and faculty demographics, and the evolution of the merged elementary/special education (ESE) program. Section Two consisted of two parts: (a) Program Design, and (b) Program Im- plementation. Program Design described the four major program themes identified by the team from the “story” as told by stakeholders. Also included in the Design Section was information about course content and sequence and description of the integration of the four program themes throughout the years of the ESE program. Program Implementation described ESE course articulation, the coordination of courses, coursework application in the field, collaboration with practicing, educators, and strategies for evaluation. Section three discussed the four major program themes from the

Both the instructional content and the instructional model of the ESE program are developmentally oriented: Faculty teach students as they expect students/future teachers to teach. The developmental approach requires profes- sors as well as these future teachers to adjust their instruction, both process and content, to the learner’s level of cognitive development, content mastery, learning style, and, when un- derstood, other aspects of the perhaps more subtle individuality of the learner.

The process through which students in the ESE program are exposed to the developmental orientation is integrated throughout the 4 years of the program. Students experience a variety of instructors’ perspectives on development and acquire knowledge about a number of developmental models. They are taught direct instruction, group instruction, individualization instruction and assessment as ways to evaluate and understand the developmental positions of their students. Faculty work to develop teachers


who are profoundly aware of individual and developmental differences among children and who make constructive use of that information to tailor instruction to the evolving learning styles, understandings, and motivations of their students.

Critical Reflection

Effective and humane teachers must in- tertwine the developmental approach with that of critical reflection. Consequently, faculty carefully design and/or select strategies which help ESE students be critically reflective. The primary intent is that students will constantly scrutinize their own teaching-related behaviors in the light of what effects they want and how they think they must act in order to achieve those effects. When behaviors work, they will be kept; when they do not work, it is expected that critically reflective teachers will alter their behavior and once again attempt to produce that effect and once again watch for the desired result. This process will continue until the desired result is produced or until a new goal and/ or behavior is selected.

As is true with the developmental orientation, faculty continually build into their courses and teaching, as well as supervision, strategies designed to nurture critical reflection on the part of ESE students. Students are initiated into the understanding that no thought is to be taken at face value, that all thoughts and conceptions must pass criteria rather than being accepted because uttered by authority or written in some book. Critical reflection is expected of students increasingly throughout the program as each attempt to evoke a teaching mantle that will be effective and consistent with developing values as well. Microteaching, journals, lesson plans, and critiques are notable as processes which have evoked reflection as a primary focus.

Grounding in Research

Distinct from, yet compatible with a developmental orientation and critical-reflection, is the presence of a literature of valid teaching effectiveness research. Faculty want ESE students to be thoroughly familiar with, and to practice, behaviors identified in the effective teaching literature. Each child, each group, and each teacher

is unique, consequently the interplay of all in a whole is even more unique. Thus, it is unlikely that any practice, even practices imported di- rectly from the research literature, may be brought into any classroom without adaptation and awareness. It is the critical-reflective awareness of ESE students/teachers which will be sensitive to this reality. So, while faculty have the expectation that ESE students’ reper- toire of professional actions will be responsive to, and often inspired by, the evolving body of practice-related educational research literature, it is also expected that teachers will make “the student” and “the class” the final measure of the worth of any approach.

Distillation of Program Understandings Through Field Applications

Students in the merged program spend in- depth time in special and regular classrooms from the freshman year on. This engagement is for the purpose of pushing academic knowledge to the level of useful classroom application. Such frequent placement in schools not only promotes application of knowledge but also provides practice in navigating within the bureaucracy of the school system. Navigation skills are considered especially important because these future teachers have been prepared in ways which are often quite different from the traditional preparation and in ways which maybe somewhat in conflict with the traditional operation of schools and classrooms. Supervisors provide consistent feedback in the field and seminar experiences constantly rein- tegrate the four major themes of the program as described here. It is especially during the student teaching experience that students are expected to engage in ownership, application, synthesis, sensitive assessment of students, problem-solving, and rapid-pace decision making. These are the behaviors which have been valued and learned through the developmental orientation, critical reflection, and grounding in research foci of the program.

Implications

This paper described how naturalistic inquiry could be used to evaluate the complex, ideog-


raphic, dynamic nature of a teacher education program. Part of the rationale for choosing this methodology was the authors’ recognition that it would be difficult, if not impossible, to truly capture the richness of what is happening in this innovative teacher education program by re- ducing it to discuss variables which would be carefully controlled.

Through the initial open-ended approach followed by probing and the somewhat more structured interviews of the second site visit, and through the reality checks (e.g., team discussions, debriefing session after the first site visit, the draft case study with feedback from stakeholders) much rich detail was collected which likely would not have surfaced through a quantitative evaluation of the program which focused only on specific, preset variables.

For example, had we started with the basic a priori assumptions of positivistic research, we probably would have focused our research questions on the stated goal of the program to produce developmentally oriented teachers. We would have described the characteristics of the developmentally oriented teacher in the classroom (which we did do) and tried to com- pare this teacher to a more traditionally trained teacher (which we also tried to do). We would also have described the evolution of the program, the course sequence and content which produced the developmentally oriented teacher, and the characteristics of the students and the faculty (which we also did). But, as we explored, through repeated interviews, observations, documents, multiple field visits, and reality checks, we discovered other more subtle but important underlying themes. For example, the emphasis on preparing teachers to be critically reflective thinkers and to act on those re- flections, the grounding of the program in recent research and literature, the manner in which these themes were applied to field experiences, the socialization of students and faculty into the program, and the interrelationships of these themes were central to the stakeholders’ thoughts concerning the program and thus es- sential to producing this new breed of teacher.

Our choice of methodology was also influ- enced by our opinion that results of quantitative research studies are frequently not realistically generalizable, particularly as they apply to the study of much of teacher education and other

issues involving teaching, learning, and school organization. The nature of the institution, the department, the faculty, the students, the schools, and the state accreditation and certifi- cation requirements all influence a program and limit generalizability. We recognize that the findings reported in our case study also lack generalizability in the conventional sense which assumes that variables are controlled, since there was no attempt to control variables. Rather it is intended that providing a detailed story of one program from one institution (which is done in the monograph), through use of naturalistic inquiry as a program evaluation tool, will allow other teacher educators interested in exploring the merger of elementary/ special education to consider the elements in- fluencing this particular program and determine which, if any, of the findings are relevant (i.e., transferable) to their own unique environment and realities.

It is also intended that the manner of presentation found here will assist readers in making decisions about the extent to which naturalistic inquiry may contribute to the evaluation of educational issues of interest to them.

Our own experience with naturalistic inquiry, as described in the previous pages, gives tes- timony to the educative, tactical, and catalytic authenticity of the method. In several cases we have found that acting “as if” the assumption set were true has created an inquiry experience which made the constructivist orientation more compelling. We have come to view the approach as one which affords fuller understanding both of the whole and the parts of human systems.

References

Fetterman, D. (1988). Qualitative approaches to evaluating education. Educafional Researcher, 17(8), l&16.

Cuba, E. G. (1981). Criteria for assessing the trustworthiness of naturalistic inquiries. Educational Communicarion and Technology Journal, 29,75-92.

Cuba, E. G. (1987). Naturalistic evaluation. In D. S. Cordray, H. S. Bloom, & R. J. Light (Eds.), Evaluation practice in review: New directions for program evaluation (No. 34). San Francisco: Jossey-Bass.

Cuba, E. G., & Lincoln, Y. S. (1981). Effective evaluation: Improving the usefulness of evaluation through responsive and naturalistic approaches. San Francisco: Jossey-Bass.

Cuba, E. G., & Lincoln, Y. S. (1988). Naturalistic and rationalistic inquiry. In J. P. Keeves (Ed.), Educational


research, methodology and measurement: An Lincoln, Y. S.. & Cuba, E. G. (1985). Naturalistic inquiry. international handbook. New York: Pergamon Press. Beverly Hills, CA: Sage.

Lincoln, Y. S. (1986, April). The development of intrinsic Lincoln, Y. S., & Guba, E. G. (1986). But is it rigorous’? criteria for authenticity: A model for trust in naturalistic Trustworthiness and authenticity in naturalistic researchers. Paper presented at the Annual Meeting of evaluation. In D. D. Williams (Ed.), Naturalistic the American Educational Research Association, San evaluation: New directions for program evaluation (No. Francisco, CA. (ERIC Document Reproduction Service 30). San Francisco: Jossey-Bass. No. ED 270 459.) Skrtic, T. M. (1985). Doing naturalistic research into

Lincoln, Y. S., & Cuba, E. G. (1982, March). Establishing educational organizations. In Y. S. Lincoln (Ed.), dependability and confirmability in naturalistic inquiry through an audit. Paper presented at the Annual Meeting

Organizational theory and inquiry: The paradigm revolution (pp. 185-220). Beverly Hills, CA: Sage.

of the American Educational Research Association, New York, NY. (ERIC Document Reproduction Service No. ED 216 019.) Received 2 January 1990 0

Date post:	25-Dec-2016
Category:	Documents
Upload:	lynne
View:	212 times
Download:	0 times

Naturalistic inquiry applied to the evaluation of a teacher education program

Documents