DEALING WITH CHANGES IN A PROGRAM'S GOALS AND DESIGN:METHODS FROM A FORMATIVE EVALUATION OF THE FLORIDALINKAGE SYSTEMGarrett Foster and Peter Easton, Fiorida State University

What role can formative evaluation play in reformulating initial program goaisand design during the development and implementation of a new program? Evi-dence from the titerature on educationat innovation and from the current experi-ence of the Florida Linkage System, a project designed to develop problem-solvingcapabitities in tocat schools, indicates that reconsideration of objectives and meth-ods in the course of implementation is a criticat need of most change projects. Butexisting modets of formative evaluation provide little guidance for such an under-taking. In this article, the need for formative evaluation of program goals and designis argued and the methods devetoped to meet this need in the context of a recenttycompleted project are presented.

THE PROBLEM. Recent literature on educational program imptementationhas taid emphasis on the changes that the goats and methods of innovative programstypicalty undergo—or need to undergo-in the course of imptementation, and onthe processes of "mutual adaptation" necessary for staff and participants to adjustthe initial program design to their maturing perceptions of local needs and poten-tiats (Berman and McLaughlin, 1975; House, 1974,1978; Kritek, 1976' Futlan andPomfret, 1977).

Evidence from the Rand study of federally-funded programs of educationalinnovation (Berman and McLaughlin, 1975) and from our own work with thedissemination and diffusion of educationai R&D (Foster et al., 1979) indicates notonly that goal priorities and appreciations of program methodology often change inthe course of implementation, but also that the perceptions of different subgroupsof project-retated personnet may shift in markedly different directions.

We have argued elsewhere that this situation has important implications forthe practice of evaluation (Foster and Easton, 1980b). It is evident that a sum-mative evaluation which assesses program worth in terms of an outmoded set ofgoals or attempts to explain resutts on the basis of an outmoded version of theprogram design risks passing very wide of the mark. Detailed descriptions of thechanges which have taken place in the objectives and methods of the program areneeded to serve as a basis for these overall judgments. Such data could be providedby systematic formative evatuation of program goals and design carried out duringthe course of implementation—and the effort to chronicte these changes wouldseem potentiatly as usefut to supervisory staff and program participants as tosummative evaluators. Yet present evaluation methodotogy does not provide uswith a clear approach for adjusting to changes in the goals and methods of a pro-gram or for helping those concerned with the program to articulate and direct thisprocess.

One method for the formative evaluation of program goals and design hasbeen worked out in the framework of an evatuation of the Florida Linkage System(FLS). FLS is a statewide program designed to assist locat school personnel in theidentification of instructional and organizational problems within their schools andin the selection and adaptation of appropriate sotutions. In this article, we describethe methods developed for adjusting the goats and design of FLS and discuss theresults and implications of their application. The following section is devoted to abrief description of the Florida Linkage System itsetf. Ensuing sections deal withthe development and application of a formative evaluation methodology for adjust-ing program goals and design during the implementation stage.

THE FLORIDA LINKAGE SYSTEM. The Florida Linkage System wasestablished in 1976 with NIE support in order to provide Florida schools withaccess to educationat R&D products adapted to local needs and in order to assistthem in developing the capacity to analyze and resolve their own instructionat ororganizational problems. The system "links" tocai schools to state and university

1152

agencies able to provide training and research products on request and is coordi-nated by the Office of Dissemination and Diffusion of the State Department ofEducation.

Each participating school selects from among its faculty and staff a cadre ofinternal change agents, known as a "facilitating team," who receive training ingroup problem resolution processes and select and implement appropriate solu-tions. "Linkage agents," located in regionai teacher education centers, maintaincontact with several project schools in order to support local facilitators, channeltheir request to the Office of Dissemination and Diffusion and help them utilizeto best advantage the training and R&D resources which the system makes avail-able.

The FLS model is based on a "linkage" theory of educational disseminationand diffusion principally elaborated by Robert Havelock of the University ofMichigan (Havelock, 1971). "Linkage theory" combines elements of the socialinteraction, R&D and problem-solving modets of educational diffusion which havebeen advanced over the last thirty years. It stresses the importance of connectingschoots with external sources of support in the R&D community and at the sametime promoting locat facutty participation in Identifying the instructional ororganizational probtems within the schoots which research products can helpresolve. The combining of these various approaches is counted on to result inbetter utilization of educationat R&D.

The FLS model prescribes a series of steps by which local schools can electto participate in the system, form their own facilitating teams, identify theirprincipal instructional (or organizationat) probtems and obtain outside supportin addressing them. Between 1976 and 1979, the system was field tested in 28primary schoots located in 16 counties across the State of Florida. Responsibilityfor directing formative and summative evaluation of the Florida Linkage Systemwas confided in staff of the Florida State University Cotlege of Education. Themethods described in this article were developed in the course of that evaluation,which spanned the entire life of the project.

METHODS AND INSTRUMENTS. A set of instruments was designed by theevaluation team for administration to staff and participants at various levels of theFLS program in order to entlst their assistance in re-examining the goals and modelof the program in the light of their intervening experience. The instruments in-cluded a goal analysis questionnaire, a model update questionnaire and a facilitatorsurvey. The process involved in developing and administering these instrumentsnecessitated a large measure of participation and reflection by project staff and wasitself part and parcel of our methodology of formative evaluation. We witi con-sequently present it in some detail before describing each of the three question-naires individually.

Instrument Development and Administration. The formative evaluationprocess comprised five steps, each involving consultation with project staff. Theinitial step concerned definition of the goals and the model of FLS. As is so oftenthe case with educational and sociat change projects, there existed no single ex-position of FLS objectives and methods which was universally accepted or used.Instead, there were a number of overlapping and more or less congruent versionspresented in different project documents. The first task for the evaluators was,therefore, to identify the documents thought to contain the essentials of FLSmethodotogy and to review this choice with project staff.

Once a coherent version of the inltlat goals and model of the project hadbeen settled upon, the instruments themsetves could be defined. Each was designedto elicit the judgment of project personnel concerning the relative importance and

1153

practicality of different goals, modei components and project tasks. The entireset of instruments was reviewed with FLS staff and field tested on a small samplebefore administration.

The fourth step invotved administering the questionnaires to the appropriategroups of respondents. Wherever feasible, the evaluation team opted for directcontact with the respondents in a group setting in order to allow for feedbackabout questionnaire content and to emphasize the "formative" character of theactivity. The goal and model analysis instruments were administered in separategroup settings to the FLS staff, and linkers, and members of the FLS advisoryboards. Only local school personnel received mail-in goal analysis questionnaires,and the nature and purpose of these questionnaires was discussed with them by thelinkers, most of whom had previously participated in a round-table group evalua-tion on the same theme.

The last step involved analysis and communication of the results of thequestionnaires. In general, data were simplified in two ways to ensure ease ofinterpretation:

1. Responses to scales of attitude or opinion (about, e.g., the importanceor practicality of different goals) were dichotomized.

2. Data were then summarized in terms of group consensus, operationallydefined as agreement by two-thirds of a given group of respondents;that is, the goal statements and model components judged "important"or "practicable" by two-thirds of the respondents were distinguishedfrom those judged "unimportant" or "impracticable" by a two-thirdsmajority and from those goals eliciting no such consensus in eitherdirection.

It was consequently possibte to calculate in relatively short time the resultsof each of the questionnaires and to feed this information back to the respondentsfor discussion and program revision. The comments and program revisions werealso summarized in a written report of results (Foster et al., 1979).

Goal Analysis Questionnaire. The first of the instruments was a "goal analysisquestionnaire," administered to those of the project staff and related policy makingbodies who were most acquainted with FLS goals and objectives: the centralproject staff, members of two FLS advisory boards, the TEC linkers, and principalsof the participating schools. The goal anatysis questionnaire listed 29 specificgoals subsumed under 7 broad goal categories, all culled from FLS project docu-ments. Some of the goal statements were overlapping, but the redundancies wereallowed to stand in order to determine which phrasing or formulation was preferred.

The goal analysis questionnaire was divided into two parts. In the first,respondents were asked to rate on separate six-point Likert-type scales, the extentto which each goal should be used as a criterion for an upcoming summative evalua-tion and the extent to which each should be emphasized in future administrationof the project. The "future emphasis" scale was added in order to elicit respon-dents' feelings about the importance of program goals independently of the imme-diate context of accountability. Space was also provided for listing other goalswhich individuals judged relevant or critical.

The second part of the goal analysis questionnaire was designed to elicitrespondents' judgment of the relative importance of the broad goal categories.Fifteen paired comparisons were used to establish a ranking among the prinicipalgoal groups.

1154

The first part of the goal analysis questionnaire was scored by the two-thirdsconsensus method described above in order to facilitate presentation of the resultsof FLS staff and participants and to clarify common-sense interpretation of thefindings. Results of the paired comparison of the major goal clusters were aniayzedto yield a quasi equal-interval ranking of these general objectives by the overallgroup of respondents and by each of the subgroups.

Model Update Questionnaire, The second instrument developed for formativeevaluation of program input was a modet update questionnaire, derived from ananalysis of project documents, which detailed the specific steps prescribed by FLSfor the identification of school-level programs, the selection of appropriate solu-tions and the imptementation of new programs. The entire exercise of devetoping,administering and interpreting the model update questionnaire was conceived ofas a formative process which would both enable field staff to articulate theirevaluations of program design and help project management and advisors to betterunderstand how the F LS model was in fact being operationalized at the local schoollevel. It was hoped that the activity would result in a better fit betvi/een the modeland the realities of implementation in the field.

After analysis of the available documentation and progressive refinementof the results with project staff, model elements were identified and sorted intofour areas of FLS activity: systematic problem-solving, internal facilitation, ex-ternal linkage and external assistance. These then served as a basis for design ofa model update instrument which comprised four sections.

In the section of the instrument dealing with systematic problem solving,respondents were asked to indicate which of the steps in the prescribed processshould be retained, which modified and which deleted. In addition, each stage inthe FLS problem-solving method was rated to determine its importance to thesuccess of local problem solving and its practicality in terms of difficulty, effortand cost of implementation.

The sections on internal facilitation and external linkage presented a listof very specific activities (e.g., "periodically stimutate interest in emerging educa-tional innovations") prescribed by the model for accomplishment within theschool or at the district level. Respondents were asked to indicate who, (i.e.,facilitator, linker, other), if anyone, should take primary responsibility for eachof the activities.

Finally, in the section on external resources and assistance, judgments wererequested as to the appropriate level of involvement of different agents (parents,district personnel, consultants, etc.) at each of the major stages of the local problem-solving process.

Only groups immediately involved in imptementation of the FLS model wereconsidered appropriate respondents for the questionnaire. These included projectstaff, district linkers and school facilitating teams. A separate survey (see below)was developed for the latter group. As a consequence, the model update question-naire was administered only to project staff and district linkers.

Survey of Local School Personnel. The third and last instrument in theseries, a "facilitator survey," was designed along the lines of the modet updatequestionnaire for administration to members of each school's facilitating team.Seven tasks which constitute the operational version of the problem-solving modelwere delineated on the basis of an analysis of FLS documents, and the facilitatorswere asked to indicate, via a series of scales whether the task had in fact been imple-mented, how practical and useful they perceived it to be, and how important thehelp of the linker and of other support personnel had been in its accomplishment.

1155

Facititating teams from 24 schools were surveyed. Actual team compositionvaried from school to school, but all teams were composed of at least one admin-istrator plus at least two teachers. The surveys were distributed and explainedto facilitators by the linkers and mailed in after completion by the respondents.Seventy-four responses were obtained from the 24 schoots.

RESULTS. We turn now to a brief examination of the "resutts" obtained bythe apptication of our methods: the kind of data actuatly produced by each instru-ment, the use and utility of this information, and the principal pitfatis encounteredin design, administration and interpretation.

Data From the Individual Instruments. As a result of the model updateactivity, the conceptual specification of the FLS mode! was modified in severalimportant respects. Based on their experience, respondents suggested ways forsimplifying the model format. Two steps in the problem-identification processwere dropped and another was incorporated into activities already specified bythe model. Several aspects of the model were consolidated to reflect better thepractice found most useful in the field.

Results from the goal analysis questionnaire were used to identify the ob-jectives which policy makers, project staff and contract personnel judged to bemost central to the mission of FLS. They thus provided one basis for organizingthe summative evaluation of the project. Thirteen of the twenty-nine goals ab-stracted from FLS documentation were deemed appropriate as summative evalua-tion criteria by at least two-thirds of the respondents. Significantly, among thosedropped were a number of objectives relating to student performance; there wasconsensus among all groups that this dimension of evaluation should be post-poned until implementation had been completed at the school level.

In addition, the goal analysis questionnaire provided very instructive dataon differences in perceptions among the various subgroups of respondents, andthis data was a source of useful feedback between policy making and field per-sonnel. For example, results of the questionnaire made it evident the TEC linkersplaced greatest emphasis on the devetopment of problem solving and communica-tion skills at the tocai level. The advisory group on the other hand, tended toemphasize the importance of R&D dissemination, white project staff gave moreweight to the creation of institutionat linkages between locat schools and resourcecenters. Discussion of these results with the respondents helped develop greaterunity of purpose in the project and increased the interest of all participating groupsin the accomplishment of goals and in the process of summative evaluation.

Data from the facilitator survey furnished a local school perspective on FLSmethods and so provided a complement to the results of the model update andgoal anatysis questionnaires and a basis of comparison with these findings. Facilita-tors' judgments of the utility of different steps in the FLS process proved essent-ially congruent with the perceptions of the other groups reflected in the goatanalysis questionnaire (though most congruent with those of the linkers). Theirresponses also pinpointed a component of the model (the "search for alternativesolutions") which needed simplification and redesigning.

Utility of the Instruments. Overati, the three surveys—the goal anatysis,the model update and the facilitator survey—furnished useful and complementaryinformation and provided a means and an occasion for critical review of the basicelements of the Florida Linkage System by those most closety involved.

We discovered, however, that we had not extended tine use of our methodsfar enough up the poticy making tadder. Whereas district and project level personnelconcerned with FLS policy took an active part in the formative evaluation activi-ties, division heads within the state Department of Education were not explicitly

1156

involved. We had counted on project management keeping their superiors apprisedof the changing complexion of FLS goals and methods; but this communicationlink turned out to be weaker than anticipated, partly because each of the state-level officials was responsible for so many projects of similar scope that they wereunlikely to give much attention to the interim evolution of FLS without moreorganized prompting. As a consequence, top tevet poiicy makers ended up reachingdecisions about the continuation of the project on the basis of a by-then outmodedversion of its design, and state support was withdrawn at precisety the momentwhen project, district and local personnet had developed a targe degree of "owner-ship" of the undertaking and of interest in its success. The lesson for future evalua-tion efforts seems to be that some form of participation by top-level policy makersin model update and goal analysis activities should be explicitly organized from theoutset.

Despite this major shortcoming in the organization of the evaluation—which,in a sense, simply confirms the importance of update and goal analysis—use ofthe methods proved beneficial for the project in several respects. First, as men-tioned above, the results brought about a reversal of the attitude that the pro-ject should be evaluated strictly in terms of student performance. The data alsobrought to light a shift from an emphasis on product utilization to one on schoolprocess and identified at the same time those process goats which were consideredimportant criteria for evaluation. These were in fact used as the basis for sum-mative evaluation. Finally, and as a side effect, the evaluation process provideda vehicle for inter-communication among field staff about the techniques foundmost effective in the course of implementation. While furnishing feedback toproject management about the adequacy of goals and model components, district-level "linkers" also shared and refined their individual evaluations of implementa-tion methods.

CONCLUSIONS. The goal analysis and model update activities described inthis article provedtobea useful means for adjusting the official objectives andmethods of the Florida Linkage System to the changing reality of the project andthe changing perceptions of those Involved. As such, they provide a rough firstapproximation of a methodology for formative evaluation of project goals anddesign.

The essential steps in this process can be summarized as follows. It shouldbe emphasized that the process must be iterative and that source documents,proposed instruments, and preliminary findings need to be checked with projectstaff at every step along the way.

1. Assemble existing documentation on project goals and model.

2. Identify personnel groups actively involved in or concerned with theenunciation of project goals and the implementation of the projectmodel: policy makers, project staff and fietd staff constitute threecritical categories.

3. Perform content analysis of documentation in order to identify thevarious goals and the set and sequence of model components.

4. Design instruments to elicit the judgment of respondent groups asto the current (or future) importance, utility and/or practicalityof the objectives and model components identified.

1157

5. Administer the instruments and, insofar as possibte, conduct apreliminary analysis with the participants so that group reactionsto the emerging judgments may also be recorded.

6. Perform further analysis of results, scoring scaled results in a mannerto enhance reliability and ease of common-sense interpretation.

7. Feed back all results to respondents in group session if possible forinterpretation and revision of goal priorities and model charac-teristics.

8. Make the written report of results and interpretation and offerthese as input to the summative evaluation and to planning andpolicy-setting personnel.

Our experience suggests that a distinction should be maintained betweenanalysis of project goals and reconsideration of the project modei and that separateinstruments should probably be used for the two purposes. Furthermore, the sameinstrument or set of instruments may not be appropriate for all respondent groups.In the FLS case, the cadre of local change agents (school-level "facilitating team")had a much more concrete and detailed responsibility for local project activitiesand much less exposure to its conceptualization than project staff and policymakers. This difference in experience and perspective suggested separate instru-mentation, but provided, as mentioned above, a control for certain sources ofbias in the results.

Our experience likewise suggests that the participation of each group ofproject-related personnel in evaluation and feedback activities must be as care-fully planned in advance as the instruments themselves. In particular, top tevelpoiicy makers need to be included in one part or another of the goal analysisand model update exercises.

The overall "payoff" for formative evaluation or program goals and designduring the course of imptementation seems to us to more than compensate theextra effort involved. Several types of benefits proved especially important in thecontext of the Florida Linkage System. Goal analysis and model update activitiesbrought to light and articulated a shift in emphasis from student performanceand product utilization objectives to an increased concern with school process.Data on this evolution in the conception of the program had important impli-cations both for summative evaluation and for educational theory. At the sametime, the mechanisms for feedback and critical review of program design affordedproject staff an opportunity to "compare notes" and refine their methods ofintervention.

Further refinements of the approach are certainly needed. The particularsof our instrumentation relate to a specific sphere of activity-state supportededucational innovation-but goal analysis and model update appear to be a genericneed to change projects, and an appropriate methodology can best be shaped withinput from a variety of settings.

REFERENCES

Berman, P. and IVIcClaughlin, M.W., Federal Programs Supporting EducationatChange, Santa Monica, California: Rand Corporation, 1975.

1158

Foster, G.R., Richardson, G., Papagiannis, M., and Easton, P., Evaluation Studiesfor rfje Florida Linkage System, Tallahassee, Florida: Florida Departmentof Education, 1979.

Foster, G.R., and Easton, P., Field-testing the linkage modei of educational in-novation. Paper presented to the annual conference of the American Educa-tional Research Association, Boston, April, 1980.

Foster, G.R. and Easton, P. "Updating the Model and Goals of an EducationalChange Program: A Third Dimension of Formative Evaluation," CEDRQuarterty, June, 1980.

Fullan, M.E. and Pomfret, A., "Research on Curriculum and Instruction Imple-mentation," Review of Educationat Research, 47 {!), October, 1977, pp. 335-

Havelock, R.G., Ptanning for innovations Through Dissemination and Utttizationof Knowtedge, Ann Arbor, Michigan: Institute for Social Research, MichiganState University, 1971.

Kritek, W.J., "Lessons From the Literature on Implementation,"istration Quarteriy, 12 (3(, Fall, 1976, pp. 86-102.

1159