DOCUMENT RESUME ED 161 885 TA 007 175 AUTHOR Byrn,. Darcie;.'And Others TITLE Evaluation in EXtension. SPONS AGENCY Federal 'Extension Service, (DO-A) , Washington, D.C. 'R.UB DATE 67 I - NOTE 119p. EDES PRICE HC-$6.01 Plus Postage. DESCRIPTORS Data Analysis;' Data Collection; DAta Processing; *Educational Assessment; Evaluation Criteria; *Evaluation Methods; *Extension Ed4Fation; Formative Evaluation; *Guides; Interviews; Measurement Goals; *Program Evaluation; Research Methodology; *Rural Extension; Sampling; Technical Reports; Test Construction ABSTRACT The authors have written this manual to aid workers in the Cooperative Extension Service of the' United States to be . better able to understand and apply the principles and methods of evaluation. The manual contains three sections,which cover the-nature and place of evaluatidn in extension work,- the evaluation process, and the uses of.evaluation results: R.thin these sections, articles involve casual, self - checking,, and do-it-yourself evaltation; identifying problems and ovjectives; the development and follow-through of a plan of work; stages and purpose of an evaluation plan; characteristics of scientific evaluation methods; collection and selection of data; evaluation criteria;- analysis of .a teaching plan,; the depth of -an evaluative study;- sources -of data; sampling procedures; interviewing competencies; constructing and using measurement 'instruments; tabulating, analyzing and interpreting Aata; preparing an evaluation\ report; and understanding and applying evaluation results. A siample outline used to evaluate an agricultural_ extension program, an exaiple.of an informal evaluation of in-service training meetings with extension workers, sources of assistance. for the evaluator in extension education; and a list of refercinces are appended: (Author /3AC) r ********************************************************************** * /Reproductions supplied 'by EDRS are the best that can be- 'lade * * from the original document. * ***********************************************************************
Transcript
DOCUMENT RESUME
ED 161 885 TA 007 175
AUTHOR Byrn,. Darcie;.'And OthersTITLE Evaluation in EXtension.SPONS AGENCY Federal 'Extension Service, (DO-A) , Washington, D.C.'R.UB DATE 67 I -
NOTE 119p.
EDES PRICE HC-$6.01 Plus Postage.DESCRIPTORS Data Analysis;' Data Collection; DAta Processing;
ABSTRACTThe authors have written this manual to aid workers
in the Cooperative Extension Service of the' United States to be .
better able to understand and apply the principles and methods ofevaluation. The manual contains three sections,which cover the-natureand place of evaluatidn in extension work,- the evaluation process,and the uses of.evaluation results: R.thin these sections, articlesinvolve casual, self - checking,, and do-it-yourself evaltation;identifying problems and ovjectives; the development andfollow-through of a plan of work; stages and purpose of an evaluationplan; characteristics of scientific evaluation methods; collectionand selection of data; evaluation criteria;- analysis of .a teachingplan,; the depth of -an evaluative study;- sources -of data; samplingprocedures; interviewing competencies; constructing and usingmeasurement 'instruments; tabulating, analyzing and interpreting Aata;preparing an evaluation\ report; and understanding and applyingevaluation results. A siample outline used to evaluate an agricultural_extension program, an exaiple.of an informal evaluation of in-servicetraining meetings with extension workers, sources of assistance. forthe evaluator in extension education; and a list of refercinces areappended: (Author /3AC)
r
*********************************************************************** /Reproductions supplied 'by EDRS are the best that can be- 'lade *
* from the original document. ************************************************************************
/ Evaluationin
ExtensionPrepared by Division of Extension Research and Training
c4,
liAuthors:''
nARCIE BYRN
JEWELL; G. FESSENDEN
FRED P. VRUTCHEY
GLADYS GALLUP
Chairman of Author Committee:
FRED P. FRUTCHEY
Editor:,
DARcIE BYRN
* Staff members of:
Division of Extension Research and TrainintFederal Extension ServiceUnited States Department of Agriculture
COPY
ENS101i
DIVISIONOF,
FagEAReli& TRAINING
J(N74EPIE L. MATTHEWS
WARD F. MRTER
J. NEIL RAUDABAUCH
LAUREL N. SAIIROSKY.
4
Icy
PiEs141, .1,14,105 a SONS, INC.TOpegA. XAMSA.4'
.44
FOREWORD
Extension workers need facts upon wliich to basetheir decisions inctheir Cooperative Extension work.The' purpose of evaluation in Extension is to helpobtain} and interpret facts. The authors have writ-ten this 'manual to aid workers in the CooperativeExtension Service of the United States to.be betterable to understand Old apply, the principles andmethods of evaluation.
Examples are drawn from and the applicationsare made to the Cooperative Extension Service, butprinciples and methods discussed have apblicationto extension type education in other parts of theworld and to other kinds of informal education.
Extension evaluatiA means the use of the scien-tific approach in providing facts as,a basis for mak-ing. decisions, drawing ..conclusions, or foriningjudgments about the orgization and conduct ofextension work. Evaluation in Extension is to theextension worker what agricultural and home eco..,nomics research is to farmers, distributors andhomemakers.
All extension workers do some evaluation in theirday-to-day work. They make 'decisions, draw con-clusions and form judgments. The aim of thismanual is to 'help them improve their evaluations'and thereby improve their' effectiveness in program
planning, teaching methods, ithe of local leadershipand organization for extension work.
"Evaluation" is a state of mind in which an exten-sion worker wants better information about hiswork and uses the information to make his workmore. effecti ive. It is an investment which producesprogress and growth in achieving the objectives forwhich an organization has been set up.
The earlier editions of the manual have beenused for inservice training purposes atCooperativeExtension workshops, conferences and summerschools. The preliminary edition (1956) has beenused in other countries. for similar. purposes andhas been translated into. Spanish by the InstitutoInter-Americano de CienCias Agricolas de la OEA,TOrrialba, Costa Rica. The manual has served asa reference guide to Cooperative Extension workersengaged in evaluation studies and to State leadersof evaluation in Cooperative Extension.
In the present edition the authors have endavor-ed to present,evaluation that will be most ustul toteachers and students of evaluation.
GLADYS GALLUP, Di;ietorDIVISION OF 'EXTENSION RESEARCH AND TRAINING
FEDERAL' EXTENSION SERVICE
ACKNOWLEDGMENTS
The manual was first p.repared by Mrs.. Laurel K.Sabrosky, Dr. Gladys .Gallup and Dr. J. PaulLeagans of Cornell University for use in the evalua-tion course at the Western Regional ExtensionSummer School in 1946 at Colorado State Univer-sity. Later it was used as training material forother evaluation courses and workshops and asreference material on evaluation.
A revision was made in 1949, with additionalmembers of the Division contributing. During1954-56 a third edition; now out-cf-print, was pre-pared and published by thg Government Printing',Office.
Since that time members of the Division whotaught courses in evaluation at extension summerschools and the USDA Graduate School have revisedthe third editien chapters and-developed new mate-rials. .hThese have resulted in the present editionof the manual.
t 0'
Mr. Meredith C. Wils6-n contributed chapters' tothe third edition and as former Director of theDivision give it administrative support. Dr. MaryL. Collings helped to develop thee philosophy andmethods of evaluation refldcted in the manual. Mr.C. Herman', Welch, Jr., Dr; Woodsoon Fishback andMrs. Amy 'cowing contributed -their suggestions inplanning the manual and reviewing the manuscript.
Many Federal, State and county extension work-ers have made suggestions for improving the pre-liminary manual and in reviewing manuscripts ofthe present chapters.
The members' of the Division wish to expreseep apprediation to Dr. Ralph W. Tyler for his
g idance zn evaluation which has beeri the basis forthe development of the work in 'extension evalua-tion.
FRED r. FRUTCHEYChairman. AuthorsCeortmittee
T
A CONTENTS
THE NATURE AND PLACE OF EVALUATIONIN EXTENSION WORK
'' Scientific Research . 3Evaluation for Everytne 3Evaluations Cah Be Improved 4
'Tolerable Error - 4Summary l'' 5Reference 5
IL EVALUATION IN EXTENSION EDUCATIONWhat Is Cooperative Extension Wo&Basic Philosophy of Extension EducationTeachingLearning Proces?Extension Education EvaluationExtension Education and Social ActionThe Extension Teaching or Program
Development CycleIdentification of the ProblemDetermination.g ObjectivesDevelopment of a Plan of WorkFollow Through on the Plan of. WorkDetermination of Progress
SummaryReferences
6'666
7
7
88
888`9
III. THE PLACE OF EVALUATION IN EXTENSION ... 10'Why Evaluate? 10What-Can We Gain from Evaluation 10Stages of Evaluation 11Purposes of Evaluation.Who Should Do Evaluation 12SummaryReferences 12
- IV. THE SCIENTIFIC APPROACH ANDEXTENSION EVALUATION 13
Custom and Tradition 13Authority 13Personal Experience flcl Intuition 13Scientific Methnif 14Characteristics of the Scientific Method 14
Factual 14Analytical 14
6 lmpar. ial 14Reliable 14
. Objective 14Science and Scientific Method 15Five Questions 15
What Do Your Want to Know? 15Where, and How Will You
Get the Information? 16Who Will Collect the Information? 16How Will the Information Be Analyzed? 16What Does It Mean? 16
An Evaluation Outline 16'Need for the Evaluation . 16
PagePurposes of the Evaluation 16
,
Questions to Be Answered by the Evaluation 16Sources of the Information ` 16Collecting the Information 16Selecting or Constructing a Record Form : 16Analysis and Tabulation of the 'Data for Use 17IntieriVeting, Reporting and r
Applying the-Findings 17'Summary 17Reference 17
0
THE EVALUATION PROCESS a
-;7. GOALS AND OBJECTIVES 18Need for Objectives 18KindS of Objectives . 18Levels of Objectives 18
Objectives of SocietyObjectives of a Specific
Organization or Group 19Objectives of the Extension Agent ... 19Objectives of the Clientele
Stating Educational Objectives . 309
Criteria for Judging ExtensionEducational Objectives 20
Steps in Setting Goals and Objectives 21Summary 21References 21
VI. THE PLACE OF TEACHING PLAN ANALYSISIN THE EVALUATION PROCESS 22
Learning Experiences 22Analysis of the Teaching Plan 23Relationship Between Rifting Learning
Experiences and Educational Evaluation 24Summary ... . . 24References 24
VII. EVIDENCE OF PROGRESS TOWARD OBJECTIVES 25Evidence - . ... 25Depth of Evalukstion 25
In Terms of Changes in, Behavior of People 25In Terms of Opportunity 25
Levels of Behavior. Change atWhich to Measure '1' 26
Factors for Deciding WhichEvidences of Behavior to Use 26 .
Which Evidences Will the LearnersHave an Opportunity to Display? 26
Will the Things You Can CountBe Adequate as Evidence? 27
What Is Your Own Facility in 1
Measuring Different Types of Behavior? 27!ow Will You Limit the Number
of Evidences to Look For? 27What Will You Do About the Time Factor? 27What Decisions Must Yot
Make Regarding Units? 28What Face Data Will You Need? .,-`28
Summary 29References 29
VIII., Ilit:Ntrit'YIN6 Pao Hip! ARE:As PqgcAND STUDY OBJECTIVES , . :30
The Importance of9Froblem IdenAfication ..:..30Selectinga Problem for. Study 10Selecting and Defining Study okiectives .... :',1
Stating Study Objectives Ti-.- 31
,Sunimary .1.- VReferences ,. > , :32
IX. Si lItt ES OF' DATA 33
Primary or Secondary SourceUseful' Data Sou rc:es' for the. Extension Worker
(;eneralHome conomiesExtension
um Mary
X. SANIN INC(3cnera,1 Definitions and Principles .,-i-
The ll'epiesentative Sample 0I..i:zt. Sampling t - 4i)A 11,a Sampling 1Grid ,Orltandorn Poit Sampling 49Group Sampling 49Stratified Sampling 42
J1
'.. ,..
1.3
1 sting the Sample
Judgment Sampling!limiting, the Population to Study Size .... 4:
(Summary 41
References -14
Xl. Mt:mons lib' (.7(1.14-..TING DATA -15
MethOds of' collecting. Data . 45 ,,.
Considerations Which' InfluenceSelectkin of Method -15
A Indy:Ai:4 of Each Method 454.The Mail Questionnaire 5Personal Interview Sthedules 46Distributed Questionnaires or Checklists. 6Croup Interview 4 47Case Studies 47
.
6
Systematic Observation Procedure 7Systematic Study of Available Records 5
Planning to Carry Out the Evaluation 48Who Will Be -Involved in
' Gathering the Information? 5When Will the Information Be Collected? -45Where Will the Study, Be Made?, 8
Involving Others in EvaluationWhen Deciding to EvaluateWhile Planning the EvaluationWhile. Evaluation Is in ProgressWhen the Eval lation Is CompletedAfterward
SummaryReferences
XIX. UNDERSTANDING AND USING.EVALUATION REPORTS
ReportsMust ConimunieateInhibitors to CommunicationOptions' and ObligationsPersonalized PresentationBrevityThree OperationsAppraiSiiig the Quality of a Report
PurposeMethodPopulation
timelinessSummary
.
CoNcLuswN. 'Why We Need Evaluation
Basic Elements of Program EvaluationPin-Pointing OhjectivesProgram Activities and Methods.Establishing.BenchmarksDetermining Program Progress
and AccomplishmentsThe Challenge
APPENDIX
A. Evaluating a Pazture Imptovement ,
Program (Illustrative Outline)B. Example of an Informal Evaluation'C. Professional Help for the EvaluatorD. Evaluation References in
Extension Service teview
4
r rev.
89.89
, . 89893.89
909090
90.
919191929292
gal
93939'3949494 ,94949495
9595
96
96969697
9797
98
102103
. .106
vTHE NAT,URE, AND PLACE
OF EVALUATIONIN EXTENSION WORK
9
4
4
`
4
"Chapter 1.
EVALUATION WHAT IT ISPrccl P. Foltoltey
1.0r
As County Agent Brown was driving hom fi'oma hight'meeting, he reflected on the meeting witha great deal,of-satisfaction. It Was afine meeting.Ile wished all, his meetingswere as good as,hiS one,
Ile recalled the hikli leVerbfinterest which per-'sista! throughout." Ile sin-fled contentedly 'as herethembered howhe haato "kill-the meeting ratherthan let it die." After the meeting many stayed on"to ask questions and talk among themselvessmall, groups., ,
5
f..
.Wbat a pleaSant contrast to-some other meetings!Mdetings like this are what make a county/agent's°.Work mean something. Yu don't mind/the late-'ness.of the hour when you.knoWybur jo'b was welt_done.and that people were' helped.
Under the glow of his satisfaction, however Mr.'Brown ikgan to ask himself some ,qUestions. "They/.-seenied enthusiastic enough, but'what'did they get;out of the meeting? Was this an educationalence for them or was it pufely entertainment? Did '-the_ meeting: bring about/any.. desired, change inthem? Or will What.did I really have in mindto acceMplish by the: meeting? . What were my -
objectives ?" , . ,
Questions like these were tnalytieal. Soil testinghad been the topic. He. hoped: they would Mt soiree .'new informatiOn'about soil testing and liming; -that
'the meeting would create favorable attitude to-'ward soil testing and Inning, and that they would
I ,test their soils and would 1:me when necessary.' 'I"'
ObviouSly, there could be no immediate evidence .that they did test their soil.,and, did.applk, recein-
:mended amounts :of, lime. ;He Would check on that'from time to time the summer. ,However,there were .several indications that they intendedto act on the infoimation he had given them.. .
He remembered how they 4`perked up their ears"when he showed-them how easy it was tO have thesoil tested, and where they could get it do.ne. Theywere equally, attentive. while he' sPoke 'about ,thevalue of liming, the effect of liming on yields, andthe relation of cost of liming- to hicreased.value ofcrops raised. -Some were .6"usy' -ma-king notes.Others raiseilquestions 'which-indicated they wereapplying, the explanation tO their own situations. ,
Several_asked him questions after the meeting.It seemed to Mr. Brown thPt these observations,
indicated-that many persons in the group.had liarn- - 41' ed soine of- thethings he was 'trying to explain. to
111, 1.. Theyohad changed in that respect. ,,,,Fromthese obsqrvations-he could Conclude that robst. oftthem had a favorable attitude, auad. fairly trong
,..;"favorable-- attitude so strong that ,many .vot.ildprobably,. do about soil testing and. lim-trig. But Piat, of poi remained to be ,seen.
tiAs mr...Browil-wzis thinking along these lies,es, he.was 'evaluating. He',.never had any course c- tr'ain=ing iw evaluation. The Nvay_hewas thinki g camenattucal. to _him, . ' " ri ,
fiAs ?2-drove.aiong hiS thoughts' moved o to 'whythis ineetingkvas so goodand Nkhsome,otif r meet-'
ings were not snood. What hid he done] his time- .-.to help bring about the changes in /14 p,eople?More sifeci.fically, whht had he done this iiNe that -
,N(,as different from what he-; had. crone/on those:,,ocCagions which lie would like to forgei? 1:
FOrIne thing, there Was;that Chart. h;e and hissecretary, had worked -on `most -of the ;4fternport.,
': ience change.d as.he'went throtigh his. e planation,1,-He remembered how the 'expressions the aud--
aided by.the chart showing yields with 'nd withoutadequake.,. liming, relative' costs and net gain. .
) ' :He recalled, alSo, 'that this .was the ,irst.time hehad 'bothered to Make a 'demonstration, When .hehad finished showing how easy it is tOake a,,soifsample, he -saw many of -them takihg,doTr the'address of the place to send the samPie.. He could ,
tell he. had the, r complete' attention' while he wascould be ap- .
uld be.hich probably"more careful '
relating how CAC rdsults of the teeplied and what.the effeCt'on.yieid
There were some - other. things,helped. For instance, he did. muplanning 'and reheafsing forthis eeting than hehad done for other meetings, He elt Mote sure ofhinieff at the meeting.' Mr: Br, wn realized 'thatall .thee 'things contributed` to, eating up a goodlearning situation ,for _his audieothers-had done would' influe,which the peopre.Would eiiang
ce. ' What he andce the degree', to
,
`JSo that is;what my districi agent meant wlenlie said that a county agent's bb is to create learn-ing experiences by -which -people can change in thedirection of educational Objectives," thought Mr.
_ Brown as he reached home and pdt away his car:
[ 1 ]
44
This s' or,, illustrates that, whether or not a per-son is ti aineci in evaluation, he makes countlessevaluations, anyway. -Whether or not Mr., Brownwas aware of it, his refleCtions on the trip homeconstitute a forth of,evaluation.
. Evaluation is a common activity in which we allengage v.,QryAlay. We finish.a meal and evaluate itby Saying-br Only thinking, how good or bad it was.We-may ,go further than evaluate the result; wemay consider what was especially good or especiallypoor about it .andwhy. The meat was tender andjuicy; but the potatoes could have been cookedlonger.
We read a book and react in some manner. Itmay be a very interesting one and we recommendit to ou friends. It may leave. us so "cold" that wedo not bother to finish it.
What Is Evaluation?, .0The purpose of these illustrations is to point out
that all of us engage in evaluation in our day-to-dayaffairs. Trifling indeed; is the action to which we donot apply the three steps in the evaluation processin some degree of intensity. First we make `some
. observations or collect some information.- Then weapply some standards or criteria to our observa-'tions. Finally, we, form some judgment, deaw,some
A -co ncht 'on or make some decision.These ree elements are involved in all evalua-
tion. It can be done so casually that we pre 'hardlyaware of doing it, such as looking out the windowto decide whether or not to carry an iniibrella. Atthe other extreme is scientifiC. research in
problemslto get information which peoplen use. .. SomeWhere in between will fall most
kinds of evaluation undertaken by extension per-. ':, sonnet .. ,. -
The better we do the evaluation job, the lesslikely we are to `jump" to conclusions which areMisleading. The purpose of this manual is tb'dis-cuss how the evaluation process can be used to aidus in draWing better conclusions and in makingbetter ,decisions inour i,vorli.'; The manual is notdesigned to make the reader"-a specialist in extten-.
.. sibn research. -Rather, it isintended to familiarizehint with the °processes of evaluation and to helphim to make better evaluations in doing extension,
a ..-
eA
(..%
evaluations (the umbrella ,decision) and scientificresearch. The difference, howeyer, is a matter ofdegree rather than kind. It lies in the difference .'in the degree to which scientific method is neces-sary in the Solution of problems:,
Degrees of anything are usually represented on ,
a scale or continuum. Disfance, can be from zero.inches to millionff of miles. There is n:b gap in thescale. TeMperature is alsO-a continuous series 4represented in degrees from.',hcit fo; cold. Otherexamples are age; young to old; ,weight, right toheavy; quality, as in adiectives of comparison, good -.better-best; strength of attitudes, strong -to 'Weakand favorable to unfavorable; skill, 'degrees ofskill; khowledge, amounts ;fro& none to much;sound, loudness,or Sourid;. and even .color, wavelengths of lightare, expressed as. degrees. .
The examples illustratelliat Many.characteristicsof people are not on an "either -or" basis; not neces- 'sarily either present or absent;'-not.either black orwhite. There are shades. A characteristic mvay',Vepartly present-and partly absent. The degrees of:acharacteristic blend into each: other on-a .scale or
''continuum with no sharp lines-of 'distinction.In order to avoid emphasiiing differences . be
tween evaluation' and research, it is more useful toemphasize similarities. Casual everyday evalua-tions can be placed at one end of the sale, and
'scientific research at the other end. All' degrees,are found between the. two extreme's.
easuarevei'ydaymint ions 0 resew!'
"Sd entific
The value of such a scale is in realizing that wecan improve our evaluations without becomiDg ascientist Other locations can also be described-on the/scale between the two extreines,,as
el ;dual Ions I el aluat vial uat Ions studies I /17:eachCasual meryday I Self-eher.king yourself I Ext, ns i" 'Smut .
.
The above locations are areas of the scale withno sharp lines of division. One blend's into theother. The first three locations may be described
"as informal evaluations, which extension agents cando theipselves..: ,..
C;sual everyday evaluations need no further ex-'. planation. They are like the first impressions ofCounty-Agent Brown about his 'meeting or the urn-'-
Work. brella decision. They are the ones we ordinarily
Degrees of Evaluation.
When we think of evaluation as a process ofcollecting information as.a basis for making'deci-sions, forining judgments and drawing conclusions,we realize it has much in common witA scientificresearch. As with evaluation,c,we do scientific re-search to obtain usable informatidn: Even so, thereis agreat difference between our casual everyday
make without rnucof evaluatioriin the decproblems.
nsiderationDi the principlessiops we make about simple,
Self-checking. evaluations represent a consciousattempt to apply principles of evaluation. Mr.BroWn did some analytical thinking as he cirbVehome from the meeting. was doing some eval-uation at a higher level. Fie was rating questi?nato himself about his initial feeling of satisfaction.
],[ 2
..
. ' 'Ile considered lo some e'xtent the information he, of "research'. under .consideration can be identifiedobtained by his.. observations and what concussions on the eolith-ilium aild misunderstanding cleared
I
he could, draw front those obseryalions, - t. . 0'-up: The design andlechniques,can then be adaptedSelf-checking includes further checking on our ' ' to what is valid and practical hi, the situation to
ordinary observations. It includes alking fu rfher -obtain th`f accuracy needed. '. ..
with othel's, writing to others '-for their judgment, .,... ..,
-A. sending mit a brief questionnaire: having one' filled ,;Evaluation for Everydneout'at a meeting and soon. :Ir. Brown could do a :4
, .,little checking asfte Went around the
7_\country , "Exte4on workers have been hearing about.
if any of the farmers had their tioirt.c!sted since the ekalitatioit during most. of their association 'Withmeeting: '. -
, . Extension. They Sometimes feel' that as,"doers" ofDo-it-yourself -evaluations involVe more planning
and application of th6 principles. of evaluation.''They are More systematically done.. more carefullyplanned and usually requite some technical help.
. ,0Each step in the evaluation' is considered: plannedand carried out with due consideration to evalua-
.. tion principle.These evaltrotions azet'hot complex and involved.
They are usually' Surveys which 'produce usableresults and which can be done easily with sometraining in evaluation or with some technicalThe best of these sometimes pasS as, data-for amaster's thesis, They blend into the next loCation
.pon the scale, `'Extension studies."The objective of a three-week summer school
course in evaluation is thal the particripants reach -,the location,on the .scale of a do-it-yourself eva0a-,.tion; that they plan and describe the ,steps in' anevaluation which, they can -do themselves in)pn.:returnifig home. ,
Ther.last twk) locatipns on the scale, "Extension',studies" and "Scientific Tesearch: may be dbscribedas formal studies.
Extension studies are more involved and cam-,
plicatect to Plan and carry out -than any of thepi.eceding location's on the seal?. They are broader,,irt scope. They require greaten attention to sound
P.-principles of scientific procedure in order .to securethd accuracy needed.' Thpses for master's degrees
, .usually thii location.
Scientific research, of course, isat. the "top" ofthe scale, involving very complex: problems and,teclihiques for getting information iiom.which con-
-4 elusions can be draWn.. Long-time and experimental-studies to determine cause and effect relationships'are chai.actvigtic of this location. The invehtion
t, of complicated techniques and prOceZhirevfot get-ting information (for doing research) falls in, thislocatIOn. The top of the location illustrated -by atomic es,earch, satellith research -or cancer
- reseneh. -
. The scale of degrees of evaluation .serves'a's ai(irriinunications device irt a discussion for identify-ing. the kind orinquiry being considered and -to beplanned. It shows whether the iliquiry is' a very,informal study or a 'very, formal study. The kind
extensions work; evaliiation ai.1 research al-9.1:ml oftheir field; that evaluation' and -research ai'e some-thirig mystical'or for persons with years of special-.lied training; that as extension teachers they y havenever done' and will never. do any eValuatioirldr
.reseairch.. .
'The continuum. of degrees. of evaluation shOwsthat there is a place 'for evaluation in .eVerY exten-sion activity. Poe. tically anyone Can do evaluationat the first tAo levels 'with. no help at .all, Aostofthein can handle a (ilt-yourself evaluation with alittle counseling and they can cooperate at theother two. levels. .
The continuum also shows. that eve .all commonlyuse methods ofinquiry to geif'informationas a basisfor forming a judgment, making a decision, or draw-ing a cOnclusionl- We can move,,up the scale and ,
do better evaluations, brit need not beeorne full-time 'risearchers at.the far end 'of thescale.
,Early in the days of the cooperation of the land:,grant.olleges and the U. S. Department of "Agri--culture in- '1Isseminating' inforniation relating to,agriculture and home, economics, it became evidentthat if ,extension 'work were .to be" effective factswould be `needed, about Its clienlele; personnel,prOgram dlanning, feachirrg methods, local:leader-ship and Other.fattp.0..-The Cooperative Extension Service began to
carry on studies which would provide informationfor making decisions abOut theconduct,of ex=tensionteaching. These studies were more scientific thancasrfal everyday-evaluations: and broader in scope.
It was recognized that since the Extension Servicewas "recommending research 'findings to improve'agriculture: and homemaking, .it must "practice
... what it preacher and study itself through scientific-procedures in improving its work'. It was also rec-ognized that if Extension carries on the study itself,sthe findings will be better understoOd and moreilikefyit,o be used.
Any -institution should plow back into its workpercentage of its 'funds to-improve 'itself and. makeits w lrk more affective.- This is the function-ofevaluation in Extension.
As the,-size anti Importance of eXtension educa- 1_tion has grown. more 'objective information wasneeded. More formal studies have been made and
e
[3]
12
more personnel been assigned to help exten-,sion agents to make ...)etter casual everyday CVailln-tint and to make -do-it-yourself- evaluations oftheir teaching:
Evaluations Can Be !mproved
casual everyday evaluations can be misleading.and often are. because of errors in observation andinteprcqation. There can be considerable differ-ence between what we see and what we think (inter-pret) we see. Great cave must be taken to distin-guish between what we actually see and what we"read into.' what we see.
When we listen. for exanple. to a person speak-ingand espeoiallywheri we read a printed page.much of what we seem to hear or see is really suP-Wied by out- memory or what we expect to sec orhear. ',Ve overlook misprints, assuming that wesee the right letters. even though the wrong letters-are actually before us. In tli;s paragraph two mis-prints have been made iMentionall:. Did younotice them?
Our minds and our biases can.influence.ouCinter-pretations and cause errors. th.oreby affecting ourjudgments. decisions and conclusions.
Time and again it has-been ,shown that personshaving fleeting glimpses of behavior Or during theexcitment of the moment have made. sworn state-ments of what they saw Oniparti only to have beenproven positively wrong by further evidence.
Casual everyday evaluation can be improved bytrying to avoid these errors. Some of the was wecan fortify ourselves and improve our evaluationsare as follows:
1. 1ievelop an attitude of mind by which we de-liberately try to provp oul interpretationwrong. Be critical of opr own interpreta-tions,Recognize our individual 'subjective bias andtake account of it in our interpretation.
3. Before drawing conclusions from a singleinstance. make further observations to deter,mine how/Widespread or consistent the be-havior is.Check further to determine 'whether the be-havivr was due simply to courtesy. to makean impression, or was forced upon the per-son. Make allowances for or discount thesethings in our evaluation.
5. Consider other causes that could have beenresponsible for what we observed and checkon these causes. More than one thing cancause an effect.
(3. Be careful not to read into the observationwhat we hoped to find (wishful thinking) andoverlook other reasonable interpretations.
9.
7. In evaluating a result or product of lAnivioruse criteria by which to judge the result orI roduct, such as a score card.
13. Check further to determiLe whether.a prod-uct made or things claimed to have beendone by the person were really clone by theperson and not by someone elsa. .
9. Recognize th-at many kinds of behavior areon a continuum, shading front one end of thescale to'another. and are not on an all-or-nonebasis.
10, Determine whether the evidence we observedis actually -the kind of evidence we 1.vott::1accept. or whether it cannot really be con-sidered as evidence.
11. Finally.iwoid Nuiping to conchisi,ons, but if%ve must act and that we nitig do manytimes) before all the evidence is' ik act uponthe best' evidence available depending uponthe seriousness Of the consequence of our
. actions.
Tolerable Error
(),By following the preceding suggestions we canimprove our evaluations. It would be unfortunate.however. if e go so far as to become perfectionists.Then .w:e wouii'. not do any evaluation bec,auseKiillevaluations possess some error.. Human beings arenot infallible and cannot follow the preceding prac-tices with' perfection. Furthermore. absolute per-fection is not necessary. and not practical.
Of course, in collecting information upon whichto base a decision, make a judgment or draw a con-clusion, it is necessary to collect "accurate" infor-mation and to make an "accurate" interpretation.But how accurate is "accurate"? No evaluation ispezfectiy accurate.
Casual eeryday evaluations, however, are citenSubject to a high degree of inaccuracy. py\improv-ing our evaluations we reduce the size of theat least to the point where we can tolerate the errorthat remains so we can make more ar,..irate deci-sions.
In measuring the length of a room, a tape Meas-ure will give sufficient accuracy for purposes oflaying carpet. or perhaps using a yardstick will beaccurate enough_ For some purposes merely "step-ping pff" the room is ac, .irate qnough.
A show of hands at meetings of dairy farmersmay be ,accurate enough for some purposes to findOut if the farmers are improving their pastures this-year. For other purposes you may need a careful'sample survey which would show how they arejmproving their pastures.
Suffice to say at this point that evaluation alwaysinvolves someerror but that better evaluations will
[ 'I]
- 1 3_
reduce the e to a point where it is insignificantfor practical use and will riot mislead us nor othersin making decisions, forming judgments and draw-ing conclusions.
Summary
The general concept of evaluation as a processof 11) collecting information, and (2) applying stan-dards or criteria in (3) drawing conclusions; form-ing judgments, or making decisions, is usefulbecause it can be applied to all kinds of situationsconfronting extension workers. All kinds of pro-grams, methods. activities' and situations in Exten-sion can be evaluated.
We do not need to be research workers in orderto use principles of evaluation. These principlescan be applied by all persons. Perfect accuracy isnot necessary; nor is it attainable, even in scientificresearch.
There is a wide range in degrees of evaluationsfroin casual everyday evaluations' to scientific re-.search as indicated by the continuum.
We often make gross errors in our everydayevaluations and "jump" to conclusions. Care .inthe use of principles of evaluation can improve ourjudgments and decisions.
How far to go in applying evaluation principlesdepends upon the amount of error we can tolerateand still have useful resultS for our purposes.
As will be discussed in the next chapter, evalua-tion is not an isolated process. It is a continuingprocess used at all points on the extension cycle.
References
1. Good. C. V. and States, D. E. Methods of Research.New York, Appleton-Century-Crofts, 1954. Pages 1-12.
2. Appendix D--Evaluation Refi2renees in Extension Serv-ice Review:
[ 5 ]
s14
Chapter IIEVALUATION IN EXTENSION
EDUCATION,1.. \'ril Raiedulmoigh
What Is Cooperative Extension Work?
1 he Cooperative Extension Service is 'a demo-cratic educafional arrangemerit among the peoplein the counties, the State/ land-grant colleges anduniversities. and the U. S; Department of Agricul-ture. It is legally referred to as Cooperative Exten-sion Work in the basic legislation, the Smith-LeverAct, which was passed by' Cpgress in 1914.
The purpose of Cooperative Extension Work. asstated in the original Act.' is . . t aid in diffus-ing among the people Of the United States usefuland practical information on subjects-relating to ag-riculture and !tonic economics and jo'encourage theapplication of the same.'' This Act further stipu-lates that . . agricultural extension work shallconsist of the giving of instruction and practicaldemonstrations in agriculture and home economicsto persons not attending or resident in said collegesin the severlIl communities.' and imparting to such.persons information on said subjects through fielddemonstration, uublication and otherwise." it iswithin the framework of thi,;. legislation, and otheracts related tc.i' education. that the objectives andscope of cooperative extension educational pro-
grams are determined..The basic administrative unit of-the Cooperative
Extension Service is the county. It is here thatedu-eational programs are planned and where extensionteaching is done. The basic objective of-the Coop-erative Extension Service is to develop the peopleinto understanding, 'effective, self-confident mem-bers of society who will have broad vision and becapable of meeting responsibilities in their homes,their communities, their State, and their nation.We in Extension express this objective in terms ofdeveloping people to the paint where they. throughtheir own initiative and ability, may effectivelyidentify and solve the various problems directlyaffecting their welfare.
Some of the major fields of educational *respon-sibility of the Cooperative Extension'Service. in ad-dition to efficient agricultural production and ad-.,- justment and home economics as such, are: Leader-ship development; community improvement; farmand home management; publie affairs; marketingand distribution; conservation of natural resources;
housing; health; and social relationships, adjust-menti and cultural values."
Basic Philosophy of Extension Education
In eoop6rative extension education, it is ackiibwl-edged that people, must be reached where they are;that is, at their present stage of educational devel-opment and experience and at thq,ir present levelof interest and understanding. It is the functionof Extension to teach people ho-w to assess theirown needs and how to solve their own probleins. tohelp them acquire- knowledge and understanding,anci to inspire them to action. The extension teach-er serves people by teaching them how to help them-selves: in so doing, he teaches people, not subjectmatter. He teaches people how to think, not whatto think.
The success of extension teaching depend's onmuch more than knowledge and understanding oftechnical agriculture and/or homeeconomics. It isequally important that the teachers understandpeople and their problems and that they know howto apply sound Kinciples of. education, psychologyand sociology.
The goal of the .extension teacher is to teach notonly desirable practices and understandings, butalso the interdependence of practices and under-standings so fundamental to achieving' satisfyingfamily and community life. This is accomplishedthrough programs which the people and extensionPersonnel work out together.
It is the extension teacher's role. first, to inspirepeople. as individuals, families and communities towork together to identify their own problems;second; to help them determiN their Objectives;and third, to counsel and supply\ technical help tothem as needed in working toward these oitj&tives.
Teaching-Learning Process
A brief consideration of education. and the ele-ments in the processes of : ..arning and teachingbasic to the conduct of extension work, is helpfulin undetstanding Extension as an educational pro-cess. as well as the place of evaluation in extensionteaching. A definition of education which has been
I.
accepted generally by extension workers and whichhas been quoted often in extension publications is:"Education is the production of changes in humanbehavior.''' It is the aim of extension' .educationto influence neople to make desirable changes intheir behavior that will contribute to better farm-ing and homemaking, to better family and corn-.nity living.
The educatiowd changes in behavior which con-certi extension teachers, and which are expressedin teaching level objectives, may be classified asfollows:
I. Interests2. ki,m1s and purposes .-3. Information*, ktowledge and understanding4. Abilities, skills, habits and, practices5. Attitudes ;Ind emonal responses
The steps in the learning process which contri-bute to the production Gf these behavioral changeshave been identified and :Studied carefully. Bealand Bohlen, in their treatment of the diffuSion-pro-cess. suggest four steps which can be summarizedin this manner:-'
1. AwarenessPeople find Out that the idea. orpractice exists.
2. InterestPeople become concerned and rec-ognize that the idea or practice has merit,
3.. Evaluation and DecisionPeople make mentalapplications of the'. idea or practice to theiroWn situations. and reject it or decide to try it.'Testing.and'AdoptionPeople try The idea orpractice and adopt it if they are.satisfied.
Extension. Education Evaluation
Effective educationsal programs provide for thesesteps in the learning or -diffusion process and con-
, tribute to production of the previously mentionedkinds of behavioral changes. To evaluate a pro-gram, evidence must be collected about these kinds'of behavior and behavioral changes in the peopleconcerned. ;Educational evaluation, then, is theprocess of determining the degree to which desiredbehavioral changes have taken place or are current-ly taking place as a result of educational effort.Similarly, extension education evaluation is theprocess of determining how well,desired behavioralchanges have taken place or are taking place as aresult of extension educational effort.
Extension Education and Social Action
Through systematic analysis cif the action lirocessof educational programs as they move forward,-distinct and related action stages are suggested.
These stages are Involved in sticessful educationalprograms regardless of .w.tic'.nr. the educationalapproach used is intensive :individual or familycentered) or extensive (group oriented). Extensioneducators are concerned about. (1) the areas inwhich there is need for change, (2) what directionthe .change should 'fake, (3) how rapidly the changetakes place, and (4) how change can be directed sothat desired accomplishments, are maximized.
There are many functions that must be perform-ed in the successful and efficient conception andimplementation of action oriented educational pro-
-. gram. These functions can be organized logicallyinto a seque-nce of actions from the inception of anidea,:or the recognition of a need or problem, to the
I, final satisfactionGf the need. or accomplishment of'the' desired goal or objective. The steps which,usually occur in such a social 'action program canbe organized into a logical time sequence model to.provide a frameworl: for a planning and analyzingaction oriented educational programs,-
All educational change. just as other socialtakes place within the contest of social systems.The beginning of all educational change is' `withconvergence of interest and definition of need, bya few people. This stage mightbe called the initia-tion stage of an action educational program.: -
In most communities or counties there have beenpreviouS. experiences ,with similar educationalaction programs which should be considered inplanning any new educational' effort. Seldom areall the social, sub-systems of a county involved inany one action program. This being true. it be-comes necessary to outline the relevant social sys-tems, or the "target system," for a particular educa-tional effort.. Initiation of action of the "soundingboard" variety, and legitimation, or sanction giving,-are important considerations in building.Jhe foun-dation for any action educational program.
The process of organizing and planning for actionin such a way that the educational need is definedand beeomes the felt need of the people, aroundwhich those in the relevant social systems committhemselves to action and set objectives or goals forthemselves, is another broad area of concern in theconduct of action educational progrzims. At thisstage, the specific series of learning experiencesthat will be needed to attain these objectives andgoals can be set up in a plan of work. A plan de-veloped in this manner, and including evaluationfor determining progress, can then be carriedthrough.
The Extensi".. Teaching or Program .
Development Cycle
In extension educational program development,five reasonably distinct stages are recognized. Fig-
[
lb
ure 1 and accompanying -description of each stagewill help clarify this total process and will furtheremphasize the importance of evaluation throughouteach extension teaching or program developmentcycle.
2. DETERMINATIONOF,
I. IDENTIFICATIONOF PROBLEMS
."
3. DEVELOPMENT OF". PLAN OF WORK
DETERMINETHE
PROGRAM
CARRYOUT THEPROGRAM
EVALUATIONDECISION
MEANS PLANNINGACTION
4, FOLLOW THROUGHOF PLAU OF WORK
5. DETERMINATIONOF PROGRESS
Figure 1.The Extension Teaching or ProgramDevelopment Cycle
..Stage 1. Identification of the ProblemThroughcollection of facts and analysis and interpretationof situations by the people working with:the exten-sion worker, the significant needs and interests ofthe people are identified. Those areas of needs andinterests wherein it is believed improvement can beaccomplished, and which lie within the scope oflegislation applicable' to the Coopeiative ExtensionService, then' become the foundation for an immed-iate and/or long-range program. This stage canbe evaluated from the.standpoint of both contentand processes on proced,ures used.
Stage 2. Determination of ObjectivesAfter theneeds and interests have been identified, the peo-pIe,.working with the help of extension personnel,can decide on their objectives. This involves adefinition of what is to be accomplished in relationto the various needs and interests, both immediateand long-time, as well as how and by whom theseaccomplishments are to be instrumented. Criteriafor this stage can be established and evalu, ions ofboth content of the objectives and preced res fol-lowed in their determination can be made.
Stage 3. Development'of a Plan of WorkAftrthe needs and interests are identified and the,ob-jectives arc agreed upon, the plan to be followedfor,putting the program into action must be devel-oped. Sound principles of social action must befollowed in developing such a plan. Involved aresuch things as determining, (1) the specific jobsthat need to be done, (2) the subject mattv that isneeded, (3) the teaching techniques to be Used, (4)
r;
the activities to be undertaken, (5) the division ofresponsibilities, and (6) the calendar to be followed.Evaluation based on established' criteria for plansof work is very helpful at this stage of the programcycle.
Stage 4.. Follow Through on the Plan of Worlr"A good plan of work presents the most effectiveway, considering existing circumstances, to i.com-plish the agreed upon objectives. An actirn pro-gram, .based-on such a plan of workand built on afoundation of sound education and social actionprinciples, requires systematic and persistent effortfrom all concerned. Evaluations can be made ofwhat is actually done in the action pr9gram fromstandpoints of both content and procedure.
Stage 5. Determination of ProgressThe extentto which objectives are being accomplished is thebasiS for determining how well a program has suc-ceeded. Evaluation of .progress helps determinewhat- remains to be done. When objectives aresatisfactorily reached, new ones can be includedin'the program and plan of work for the followingyear. If the objectives are not accomplished satis-factorily, a revised plan of attack on the sameproblem may have to be carried over into the new_program. In addition to data for future or followupprogramming, progress evaluation also providestangible and objective data for use in annual andother periodic reports.
ActiVe partiCipation of the people in all fivestages is the core of the teaching-learning processin. Extension. f Each of these stages can provideMany experiences for those involved and it isthrough such experiences that learning takes place.
As indicated, each stage can be evaluated. Thisis important because the accomplishments and suc-cesses att?ined in any one stage are indicative ofhow much is likely to be attained in each subse-quent stage of the extension teaching cycle.
Summary
In this chapter we have stated that the Coopera-tive Extension Service was brought into existencein 1914 to perform an educational function for per-sons not in residence at land-grant institutions inthe fields of agriculture and home economics.througli the years it has developed an educationalphilosophy which calls for reaching and workingwith the people where they are in matters of educa-tional, occupational and socioeconomic maturity.Emphasis always has been on how rather than whatto think. ,
We have said that the teaching-learning processconsists of producing desired changes in humanbehavior and that it is a dynamic or action oriented
process.' We have said that evaluation is a- keyconcept in 'jle process and have outlined the fivestages of the extension teaching cycle with nota-tions as to how evaluation fits into each stage.
Next we will go into greater detail as to'whatevaluation is and why it is so fundamental to allaspects of Extension.
References
I. Association for Supervision and Curriculum Develop-ment. Guidance in the Curriculum, 1955 Ydarbook.National Education Association, Washington, D. C.
2: Beal, G. M., and Bohlen, J. M. The Diffusion Process.Iowa Agr. Ext. Serv. Special Report 18, Ames, 1957.
3. Brunner, E. deS., and Yang, E. H. P. Rural Americaand the Extension Service. New York, Teachers' Col.lege. Columbia University, 1949.
4
u.
4. Castor. N. 'The Teaching7,Learning Process. New York,Dryden Presses, 1953.
5. Deese, J. The Psychology of Learning. New York,McGraw-Hill Book Co., Inc., 1952.
i. Hannah, J. A.. tChr.) Joint Committee Report on Ex-tension' Programs, Policies and Goals. Washington,D. C., U. S.. Dept. of Agr. arid. Assn, of Lund-GrantColleges and Universities,;1948.1:7. Asey. L. D., a Hearne, C. C. Cooperative Extension')Work, 2nd Ed. thaea, Comstock Publishing Associates.CornellUniven4ity Press, 1955.
8. Lancelot. W. H. Permanent Learning. New York,aohn Wiley and Sons, Inc., 1948.
9. Miller, P. A., (Chr.) The Cooperative Extension Serv-ice ... Today: A Statement of Scope ant; Re0onsibility.Subcommittee on Scope. and Responsibility' of the 195'7Extension Committee on Organization and Policy.
10. Tyler, R. W. Basic Principles of Curriculum anci In.-struction. Chicago, University of Chicago Press..1950.
11. Wilson, M. C., and Gallup, G. Extension TeachingMethods. U. S. Dept. of Agr., Fed. Ext. S2r,... Cir. 495,Washington, D. C., 1955.
.1a
Cha terTHE PLACE O EVALUATION
IN EXTENSION
Joseph L: Matthezcs
Our objectives in this chapeef are to help thereader to: (1) understand the role of evaluation inExtension, (2) know some of the benefits that canhe gainid through systematic evaluation, (3) knowwhat can be evaluated, and (4) feel that it is impor-tant to evaluate Extension work. While so doing,we hope to provide the reader with a usable answerto the question, "What is the place of evaluationin Extension?"
We evaluate so that we can have some measure'of the desirable, and undesirable, outcomes of edu-cational action that we take to move a person orgroup of persons towatt4 desired extension goals orobjectives. We are usiltg action here in the senseof attempting to chaitgq i,ttman 'behavior in de-sired ways.
Why Evaluate?
No doubt, this, question often has entered yourmind. You may already have your answer, but wewould like to discuss oun briefly. As we stated inthe opening chapter, everyone does a certainamount of evaluation, sometimes consciously butquite often unconsciously. We ask ourselves, "How'am I doing?" at practically every stage of our work.
,.Every job we do (writing a news story, giving a talk,making a farm or home visit) brings up this ques-tion.
We may not frame the words mentally,-but afterwe have done something_we may feel good about it,or we may be only moderately satisfied with it,or our reaction may be one.r.of dissatisfaction withwhat we did. By these reactions we haVe passedjudgment on our activity and have found it goodor bad. Making judgments in this ma er a outthe value of what we have ,done is a ki d of ev lua-tion.
On a slightly- Higher plane is the kind. of .tcvali.tion we do by-assessintwhat others say or do aboutour work. If a person who has att:.-Aided-one ofyptir meetings. tells ytiu that he or she enjoyed themeeting, you goo . If several persons tell youthe meeting was a good one, you feel even betterabout it and think that it must haVe been a prettyfine meeting.
Expressions like _these .have some value as indi-cators of success. When nothing better can be
done about evaluating our work, we come to dependa lot on this kind Of evidence. However, we mustnot depend entirely dn such expressions from peo-ple who participate in our extension activities.Quite often, persons making judgments of this sorthaVe no real understanding of what the educationalpurposes of Extension are. For this reason, theyhave no sound :basi9 for making judgments aboutresults.
Sound evaluation always -must be in terms ofwhat' we set out to dothe objectives we have inmind, Therefore, expressions of the people involwed, as well as all other measures, must be appraisedin the light of what we expected to accomplishwhen we planned the educational activity
addition to-the danger that what people tell usmay not be relevant to our purposes, there areother reasons why such evaluations must be supple-mented by evidence from other sources. Personswho have participated in our program and who areour friends are reluctant to be critical even\ whenasked to be. This resultS from not wanting _t,b hurtour feelings and from an inability to separate theirfeelings toward us from their objective appraisalof our professional activities. -Later in this manualwe will have more to say' abwit the complicatingfactors of the hurnan element in scientific evalua-tion.
The preceding paragraphs lead logically to thisstatement "We all evaluate our work in some wayso why nit do it 'well?" If the first part of thestatement' is not true, then experience would have
Nno value and there would be no improvement onthNob as a result of experience. Doing evaluationwell \requires considerable self-discipline in apply-ing jcientific methods, collecting and analyzingfacts, and making judgments about their implica-tions. This manual is all about dothg valuationwell by applying scientific methods to evaluation.
What Cdn We.Gain from Evaluation?
'The list of benefits we can' gain from evaluationcould include many more than the five importantfines that we will mention here. First, extensionworkers have to make endless decisions and then,act ;:ccording to what they understand to be themandates of their decisions. Far example, a county
extension worker may -believe that appointing thechairman of an action. committee will result in bet-ter committee work than when the committeemembers choose their own chairman. Much later,and probably by accident, it may be discovered thatthe expected result actually-did not occur.
The longer Ita practice has been followed, theharder it is to be objective abut its limitations, andthe harder it is to get at making needed changes.One reason for this is a predisposition to .see theexpected results even though they are not actuallyin evidence. Preconceptions about results oftenmake perception of actual happenings so selectivethat unexpected or undesired ones go Unnoticed.Evaluation is the method of trying consciously andobjectively to find out whether or, not certainactivities -actually did lead to the results that wereanticipated.'
Good evaluation is essential when it comes tomaking periodic checks on the effectiveness ofeducational programs to decide where .improve-ments are needed.- These checks serve a functionsimilar to the balance, sheet in a business enterprise.When the sheet reports results that are 'less thanexpected. owners or managers of the business lookover the units which are. not doing well to seewhere changes are in order. In either case, deci-sions so basic to successful functioning of theorganization must be founded on accurate measure-ments and .observations, or changes may not bemade in the best direction. Surely we .all agreethat change for change's-sake is not good long-runaction,
A third benefit of good eraluation is the feelingof security it can give the worker who is doing agood job. Extension workers can b :come so involv-.ed in their i;vork that it is easy for them to worryabout their effectiveness and to have doubts as towhether they are really accomplishing their objec-tives. This concern is all right if it leads to carefulappraisal and constructive measures to improveprograms. However, when systematic evaluationsare not made, the tendency is for staff members tofeel even less secure and to retreat to activities thathave Sul-face indications of tangible results, al-though they may be less important to the work thanothers which they abandon..
We may seek security by emphasizing extensionactivities to which people will respond in large num-bers rather than .others that .could result in basiceducational changes in the long run merely becausewe 'Teed the immediate -assurance of favorablepopular reaction. Scientifically done evaluationwill give equally convincing assurance when goodwork is being' done and we know the criteria will bevalid extension Objectives rather than popularappeal:
A.-fourth benefit lies in the area of public.- rela-tions. There is no better basis for sound publicrelations than a careful and .comprehensive evalua-tion. Rather than superficial impressions or value-clouded opinions, it can provide concrete evidencethat can be adapted readily to the needs of peopleworking in the various communication media.
These folks are happy to help inform the com-munity of the contribution of your educationalprogram when they see the facts. They do not havethe feeling they are "going out on a limb" for youwhen they are handed the ammunition to deal withcriticism. They know that critics can be convertedto constructive cooperators when. concrete evidenceof accomplishment can be produced.
A final benefit we will disluss is the mitter ofwhat evaluation does to the :profe*,:Nnal attitudeof the worker whO uses scientific eluation tech-niques. In the process orstatinghis objectives insystematic fashion, specifying the desired. changes,analyzing the teaching activities designed to accom:dish the objectives, and appraising the results andprobable need for further action, he is very likely
-to adopt a more scientific approach in future under-takings. Inevitably, he will adopt more objectiveand systematic ways of thinking about his woric andhis accomplishments.
Stages of Evaluation
We can think of .evaluation as having two mainstages. Early in the program weem study arrange-nients and procedures for doing Extension work.Suppose we refer to this as means evaluation. Eval-uation of means is especially useful in the earlystages of an activity or program before Alias beengoing long enough to accomplish the desired educa-tional results. We can study methods, or means.and decide whether they are worthy of the programin which they are being used.
Examples of means evaluation would includeappraisal of the organization of people for extensioncommittees, clubs, spe6ial interest for commoditygroups and the like. Educational, niaterials usedsuch as bulletins, leaflets, newspaper articles, writ-ten programs, and plans or work are Includedamong the means which should be eyaluated,4 TheproCedures for planning programs and conductingclub meetings also should be considered. -
Evaluation in terms of means alone is hardly ade-quate, however, and should be thought of as partof the evaluation sequence rather than the wholeof evaluation. Insofar as the extension program is .
concerned, the end products'of extension work, theeducational changes that result from extension
t4teaching, are what must eventually be ev luated.We must make ends evaluation. We need o assessthe changes in knowledge, understanding, attitudes,
1".".
ills and.abilities of the people who are affectedby Extension educatiohal activities.
Purposes of Fiyaluation
In addition to kinds of evaluation in terms ofstages of the prOgram, we can think of evaluationin terms of objectives. We can think of it ii. termsof pure fact finding or measurement; or we canthink of it as a basis for interpretation. In the firstis involved matters of assembling measurable dataon changes preSumed td result from the teachingdeviceobservable behavior changes as well aschanges in material things. In the second we makeconsidered judgments about.. the data that were'collected by means of the measuring techniques.
Although most of us 'can recall when we stopped, at the first stage, or when we observed others stop
there, we,,should have no difficulty in acknowledg-ing that the evaluation mission is not accomplishedunless we face up to the interpretation Atsome point judgment must he made about thede-gree of success or lack of success that was exper,enced in the program.
Who Should Do Evaluation?
Evaluation. insofar as possible, should be doneby the extension workers who are actually respon-Sible for the program that is being evaluated. Itimproves their understanding of the program andtheir work in general. In using the methods ofscience to evaluate their work, they accumulatereliable evidence which helps them define futureproblems more sharply. They draw on these ex-periences and findings for action hypotheses thatgive promise of helping to reduce or eliminate diffi-culties in their work. They test out promisingprocedures on the job and accumulate_ evidence oftheir effectiveness in the next evaluation sequence.Eventually,' after obtaining sufficient evidence ofeffectiveness, they feel justified in generalizingwithin limits and can turn to other objectives.
An important psychological consideration inevaluation is the fact that the people who standthe best chance of benefitting professionally fromthe experience of the evaluation are the ones whodo the *evaluation. They can see at first hand theeffects of the methods and,procedures and can getspeedy answers as to how good new and promisingways of doing the job really are. For this reasonpersons responsible- for the program ,..should makeevery effort to do the evaluation or be a part ofthe evaluation team.'
Another consideration in evaluation is the prob-lem of communicating results to the ones who canuse them in th work. The communication prob-lem is solved :roportion to the degree of partici-pation by the people,wh' are responsible fdr theprogram..` When the responsible persons are fully
involved, or when they take Vital responsibility forevaluation, the communication problem virtuallyis eliminated.
Sometimes, however, the complexity of the prob-lem being studied or the technical competenceneeded, may make it impossible or impractical forthe extension worker to conduct the evaluationeffort independently. Nevertheless, drawing upohthe skills of others for assistance in evaluation neednot impair communication nor minimize the valueof the experience to the extension worker whosework is being studied. Experience has shown thatthere are some things that extension workers haveneither the time nor the technical competence tohandle by- themselves. These, very properly, areleft for State extension research persohs and forthe resident teaching or experiment station re-search staffs to deal with.'
Summar:),
In this chapter we have attempted to impress our-readers with the importance of evaluating our workon the most scientific level the particular activitywarrants. We have listed and elaborated on fiveimportant benefits to be gained by evaluating. Wehave mentioned that there is both means and endsevaluation, and that lloth have a place in extensionteaching, but that « must not stop after doing theformer when it is possible to do the latter as well.We have stated also that evaluation can be a matterof ,measurement alone, or it can be carried to itslogical conclusion by interpreting results of themeasurements to determine effectiveness of theaction that was evaluated.
Closing remarks were devoted to explaining whyextension persons responsible for the programeither should be the ones to -evaluate it, or shotildbe as closely involved as possible in the process.At the same time,, they should feel free to call onprofessionally qualified counsel and leadership insituations calling for more facilities than are attheir own disposal.
In the next chapter we will spend some timediscussing what is involved in scientific methodol-ogy and what is necessary to consider before doinga scientific evaluation.
, 05
ReferencesCorey, S. M. Action Research to Improve School Prac-tices. New'York, Columbia University Press, 1953,
2. Matthews,' J. L. How Are You DoingIn Farm and,Home Development. Talk given at the Southern Region-al Farm and Home Development Workshop; Rock Eagle,Georgia, September 21, 1955.
3. Smith, E. R., Tyler, R. W., and evaluation staff. Apprais-ing and Recording Student Progress. New York, Harperand Brothers, 1942.
4. 'Tyler, R. W. Organizing for . Using . . . Research inExtension: Summary of an Extension Research Work-shop. May 6-10, -1%9571 Washington, D. C., Federal Exten-sion Service, U. S. Department of Agriculture, 1957.
1.
[ 12 ]
2i
Chapter IVTHE SCIENTIFIC APPROACH AND
EXTENSION .EVALUATION
Fred P. Friitchey
People are curious. They have always-been curious. They. want to find out what makes things"tick." They want t(luhderstand themSelves, otherpersons and the. world,in.whieh they live.- They tryto study arid investigate puzzling questions.
This curiosity is not confined to adults. At avery early age .people are curious. The "why"questions of children are well-known to 'parentsand teachers.
Peopleare not only curious but they also haveproblems and must make decisions. Very early inthe. history of man, the problem of measuring timeWas recognized: Man put his ingenuity to work tofind a solution.. The, measurement of time pro-gressed over the years from such gross' units asthree moons ago, through the hour-glass sta e toelectric clocks and-self-winding watches.
How do people find. out things? How do ey,make up their minds? How do they know what tobelieve, 'NM what-td do?
.
During the history of man's deirelobment,have used a variety of approaches to dain under-standing and to solve prohlems. They have usedcustom and tradition, alit .iority, personal experi-ence and intuition, and the scientific method. . '
Custom -.and Tradition
We use custom and tradition many- times in ourlives in reaching decisions,And with no harmfulaffects, as.in ing clothes, and many times forhighly essential ai'd valuable purposes, as for ex-ample, courtesy anti manners.
But when we use custom and tradition for all ourdecisions, improvement and progress out thewindow." If 'farming dpcisions were dictated en-tirely by the farm practices of our ancestors. exten-sion agents would have a difficult time implemetpt-ing the findings of the experiment station.
e would have no automobile,-no radio, no elec-ric power,' no telephone and4a host okother things
because they were not our custom or tradition. Thepast would be perfectly preserved.
What would happen if extension agents made alltheir decisions on.the basis of .custom and tradition?
Authority
Custom and tradition did not always, provide asatisfactory answer to curious people; nor solutions
' to urgent problems. People appealed to "authori-ties," those in whom they had confidence and whomthey thought should "know."
They went to the "wise men of the tribe," to theoracles, to the "medicine.men" and up through theages to the .authorities of today. Much of ourteachin:g in schools and colleges is accepted onauthority. We cannot possibly explore and com-prehend all knowedge. So we dependson author-ity
People frequently consult tech nical- and profes-sionar authorities like physicians, County extensionagents, teachers, lawyers, bankers, and so' on.
l'ootball players are governed by the rules of thegame. Referees make their decisions on the basisof the rules. Card players are governed by therules of the game. Extension workers are governedby the policies of the institution.
But authority must get its knowledge in someway and arrive at the rules of the game, the policiesof an institution, or the advice of experts.
[ 13 ]
Personal Experience and,Intuitio6=
Personal experience is a widely used basis fo'making decisions. Notice how often it is resatedto in argumentative situations, and how it is givenas a reason for -our decisions. This is done con-,.sciously.
But intuition is soawhat different. Intuitiveinsights seem to come from nowhere. They arehunches. Have you ever worked long and hard ona problem and have 'finally gone to bed withoutsolving it? Upon. awakening in the, morning, thesolution came to-you like a flash. Where it camefrom is difficult to say. It just came.
Or have you ever had to make a judgment whenyou know what it should be but could give no goodreason for it? These are two examples of intuition.Such decisions or judgments seem to be 'based onpersonal experience and to have come out of thesubconscious.
Long practical experience in a field of work tends.to develop this intuitive power in u.,3 and to make it
easy for us to use intuitive judgments. They arefrequently right, and they are, frequently wrong.
Personal experience can be vei'y misleading attimes. For example, we may draw a conclusionfrom a single instance. Note in Chapter I other,common errors we, frequently make during per-sonal experience in getting information and form-ing judgments.
Following the rules of evaluation will help us touse personal experience as a morezatisfactory basisfor making decisions and checking 'onqiur intuitivejudgments.
Scientific Method
Toward the end of the 16th century another basisfor making.decisions was tried. This was checkingtii see, testing it, experimenting with,.'7it, trying itout:
The often quoted illustration -of_the use of the.scientific qiethod was Galileo's experifnent. InItaly in 158:9Wlileo and others were arguing aboutwhether a heavier.weight,Lalls fagter than a lighterweight. Some .said that it stands.to reason that: aheavier yejght will fall. faster.
Galileo proposed that they drop twnweights, oneheavier than the other, from the leaning tower ofPisa at the same-time and see which one hit theground first. This was a. most unusual way of de-
siding a question at that time.Eventually Galileo tiled this way. Both weights
when dropped at the same time reached the groundat the same time. They dropped with equal speed,neglecting air resistance. What an odd way to"prove" anything; This was a revolution in think-ing. it opened the way for discovery. It freed themindS of people to find' out new things.
Today we take Galileo's method -as an acceptablemethodor do we? How many times do we makedecisions, for'm jUdgments, jump to conclusionswithout checking to obtain a better basis for ouractions?
The method used by Galileo is the method whichhas made it easier to understand ourselves and theworld in which we live. It is the method whichhas,made possible the progress of today. it is themethod of evaluation.aCharacteristics of the Scientific Method
There are .certain characteristics which identifythe scientific method of making decisions, fbrmingjudgments or drawing conclusions. We will discussfive of them briefly at this point and elaborate onthem in succeeding chapters.
1. FactualThe scientific ethod is a way ofw,orking by which observable and demonstrable evi-dences can be obtained and upon which people can
[ 14 I
'agree.' If we want to find out whether farmers areadopting certain practiceS or have certain attitudes,we can do as Galileo .didlook and see,
2. AnalyticalMost brfe is very complex andinvolved. It must be oken down into its compo-nent parts so that the human mind can deal with it.Most processes in Extension are too complex tostudy as entireties.
'Even leadership, a component part of extensioneducation, is not manageable until it,is analyzed tothe point where there are "handles" that we canlake hold of and understand. The big intangibleproblem must bj broke,n up into sing/1 more tan--gible problems.
Evaluation may seem to be a very complex pro-cess. Through analysis it can be,brokendown into.its componEnt aml'understandable parts.. Later inthis chapter steps in evaluation are presented andsucceeding chapters amplify these 'steps.
3. ImpartialScientific method is-a method ofattacking problems. Scientific attitude is 'how_we-feel abo,ut using scientific method. It is a disposition or willingness to apply .scientific thinking.
A debater argues to prove his side of an issue. Hecanot'have a scientific. attitude. If he did he mightweaken his argument, and 'lose his case.. He darenot preSent both sides Of the issue. Tire opposingdebater presents the other side.
A' lawyer, like a debater, cannot argue for bothsides. He supports and. .str;-.ngthens his -own side'of the case only.
A judge or jury, on the other hand, must have ascientific attitude. They inusts look 'at both sidesand weigh the pros and eons of evidenceimpartially.
A scientific attitude is a willingness to look; at ourinterests, prejudices, and loyalties. It .i3 aness to examine and re-examine our beliefs andexplore new ideas.
A scientific mind seeks what is just., true \andbeautiful. It is alert, curious, interested and a littleexcited about the world. It is an open mind.(notto be confused with an empty mind). It is an atti-tude of, mind that is learned; not inherited.
4. Reliable Have you ever' noticed that just be-fore Chlistmas children are on their gbod behavior?During the rest of the year they are just normalchildren. If you observed those children only justbefore Christmas or only at some ,other time of theyear when their behavior was at low ebb, youwould come to opposite Conclusions. '
Reliability means that we should have a sampleof representative observations and be careful aboytdrawing conclusions from a single observatior :htany particular time.
5. ObjectiveEach of us uses our ,'subjectivejudgment in interpreting what we see and in draw-
40
ing conclusions about our observations:- Do otherss6e what We see'and do they draw the same conclu-sions? The degree to which they,do so is objectivity.'. If we look at a clOg, each of us would agree that
a clog. "Except perhaps at a clog shdw, we might...see 4n entry and ask, "Is that a dog?"
:;:,jectivity is the agreement 'among people of.what they observe and of their interpretations. Inevaluation we. are concerned about agreement onthe evidence we are looking for and Upon recogniz-ing it when we find it.
We are cqncened about agreement and under-standing of the methods used in evaluation. p Weare concerned about agreement ofi",the- interpretation and conclusions reached..
Many arguments result from differences of opin-ion. The use of scientific thinking helps to dispelthese heated arguments. When people clarify what..they are -arguing about. they often find that theyare not arguing about the same thing.
Have you ever been in a CliSCUSSiOri where afteran hour the group found that all were not talkingabout the same thing? Clarifying the questionmakes for better communications. more efficientuse of time and better human relations.
The above examples illustrate the need Of objec-tivV to assure understanding, of the question 'atissue.
Objectivity- is increased by apply ng numbers toour observations. Whereas "a few ersons" maymean 3 or 10 persons. "5 persons". is ery definite."Five persons" V' ore objectivethan "a few per-sons." All 0-f Us can look and.see 'if there are fivepersons.
AL one time length or distance was not expressedqualtitatively. Such expressions as "a piece way.dot n the roads' or "quite a long way from here"wee used. Nov we measure distance in the num-ber of inches. feet. yards, miles.and'even light-years.
The thermometer is a more objective measure of-heat than our feelings of being hot or cold. It hasbeen said that progress in scientific Study hasaccelerated with the development of-satisfactoryquantitative measurement's. These are used exten-sively at the upper end of the scale of degrees ofevaltr'
11L things in the behavioral sciences today.however, cannot be quantitatively measured; theycan only be described. As time goes on, and bettermeasuring devices are built, more quantitativemeasurements will be made.
Science and Scientific Method
To some people research is associated with 'sub-ject-matter fields.' Science is considered to be inthe fields of chemistry and physics. To these per-sons theliolegical fields such as medicine: plants,animals and food are considered on the fringe of
,science. They be,cation, psycholosimilar fields alscience.
n
ieve the social studies like edu-y, sociology, anthropology, bande clearly outside the realm of
Other person, claim that all of these fields arein the realm.of,rciencd. The difference of opinionraises the qt.icStion, "What is science?" Is sciencea field of subject Matter or is it a method of think-ing? To the latter group of people, physical, bio-logical, o/social. phenomena are not in themgelvesscience the human mind brings to bear on/these phenomena its powers of observation, analysis, synthesis, and verification in organiting thesephenomena for use by people.
In this sense, science as a method'of thought hasapplication to all fields of subject matter. Scientificmethod involves the logical thought processes basedon the recognition of assumptions and definitions.It involves the collection and interpretation of dataand the control of factors which might influencethe result of an experiment, a study or an ifivesti-
egation.Mathematics can be applied to all the fields, more
accurately to some than to others, In similar man-ner experimental Controls can be applied to all thefields,' With much more difficulty in some. than inothers.
The past 50 years have shown marked progress.in the application of scientific method to problemsin the field of the social sciences. Evaluation inhuman behavior'has shared in this progress. Todaywe have evaluation in extension work as an appli-cation of scientific method with various degrees ofrigor depending 'upon practical conditions andusefulness of the results.
Five Questions
We have found that by asking ourselves fivesimple questions we can get off to a fast start on a
'problem requiring evaluation. Slight modificationswill make them fit most situations.
1. What Do You Want to Know?--The first thingyou need to do is decide emactly what, you want tofind out. What are the questiOns you want answer-ed by your evaluation? Work ,out these questionsas -carefully as you can. because they determineanti guide the conduct of the evaluation, and be-cause the answers will help you in your- work.
The 'questions will guide you in stating the speci-fiequestions for the questionnaire or other devicesyou use. Analysis can be made of the replies youreceive to these questions. The report of yourevaluation can be organized around these ques-tions. In 'fact the report will be the answers tothese questions.
By carefully ,determining what you want rt,i)
p
know, an initial, vague, general feeling of concernan -be broken down into questions which you can
tackle.
2. Where, and How Will You GO 'the Informa-tion?--This step has.two parts; where to get theinformation, and how to get it.
Where means from whom or from what othersource. Will it be from 'adults; from men, from allfarmers' in a county, from dairy farmers, or fromGrade A milk dairy farrpeirs? Will it be forri allpersons in any one of the groups or from a sampleof persons in -one ofthe groups?
Other sources of information are records andreports, such as annuriports, the census, writtenprograms, minutes of meetings, and so orf.
How you will-get the information refers to mailquestionnaires, observations, interviews, tests,check lists, score cardS., case studies, and so on.Forms arc prepared on which the information canbe recorded. They are used to facilitate handlingthe data and so ::ou do not have to depend upon
, memory.When you have thought through and know the
information you want and also where and how,youget it, you are well on your way in planning
an evaluation.
Who Will Collect the Information?.The nextstep is to decide.,whether you or a group of 'otherpersons will collect the information. What training
1Will be necessary if you, have, others conect theinformation?
4. How Will the Information Be Analyzed?Nowis the time., to goback again to the first steptheinformation you wanted. The replies on the recordforms are organized and summarized to answer thequestions formulated in step one.
5.' What Does It' Mean?What are applicationsor the results to extension work? In what way canthey be used to make our work better? After know-ing these results, what changes should one makein his way of doing extension work?
Oftt, the changes are not big revolutionary-changes. Many times there are little ways, such asdoing something different in organizing a group,or in writing-a letter, or in a faym or home visit.You may get to know your group better and under-stand why some do not attend meetings or whythey are not interested in new practices.
All of these Steps are thoroughly discussed insucceeding chapters.
An EvaluatiOn Outline
The five questions we have been Acussing canbe expanded into an evaluation outline as a plan of
[167
action that yOu can use in making "do-it-yourselfevaluations" and "extension studies." elf
Although the steps have a certain element ofsequence to them, it is wise to keep all steps prettywell in mind while working out the details of anyone of them. All the steps are not usually taken inany given order. Never consider only one step ata time without thinking about the other steps, since.all steps are related to each other.
1. Need for the EYaluation \ca. What extension project,' problem, activity,
job, method or situation do you want toevaluate?
b. Why-evaluate it?c. How can the results of the,..evaluation be
used in your extension work?
2. Purposes of the Evaluationa. Is the evaluation,
(1) an analysis of a situation, or(2) an evaluatiOn of teaching objectives?
b. What questions should be answered by theevaluation?',
3. Questions to be Answered by the EvaluatiOna. If analysis of a situation, clarify the kinds
of information needed to answer the ques-tions.
b. If an evaluation of teaching, clarify theteaching objectives. Analyze the teachingplan.
)
(1) Review what has been taught.(2) How it was taught, and to whom.,(3) Changes in behavior expected as a re-
result of the teaching.
4. Sources of the Informationa. Peoplefarmers, homemakers, local lead-
ers, club members, non-members, extensionagents, etc.(1) Do you need a sample of these people?(2) How will you draw a sample?
b. Recorded information reports, census,etc.
5. Collecting the Informationa. How? Interviews, mailed questionnaires,
observations, etc.b. By,whom? Person making the evaluation,
extension personnel, local leaders, etc
6. Selecting or Constructing a Record, Forma. Kind of form; questionnaires, interview
forms, tests observation sheets, ratingscales, check lists, score cards, anecdotalrecords, etc,, to be used.
b. Data about(l the siluatio to be studied,'2) evidence of progress toward the teach-
ing objectives, or(3) "face data" to be collected.
c. Formulation of questions or statementsd. Physical set -up of the record\forrne. Pretesting and revisions
7. Analysis arid Tabulation of the Data for Usea. Classification and sorting of data to answer
questions in Step 2 b.b. Work tables neededc. Coding of free-answer questionsd. Method of tabulatione. Sorts and strbsorts to bring out relation-
ships
8. Interpreting, Reporting and Applying theFindingsa. Preparation .of- tables, charts, and graphsb. Summary of findingsc. Applications w extension work.
Summary
The curiosity and necessity of people to learnabout themselves their environment has occa-sioned them to various methods ogetting infor-mation and making decisions.
The most satisfactory method has been the
7
[17)
scientific method. It is the method of careful obser-''vation and control of subjective bias. Scientificmethod can be used in solving all kinds of problemsand is not limited to the physical sciences.
Five simple questions in evaluation are:
1. 'What information dO'' you want to get?.2 Where, and how will you get the information?3. Who will collect the information?4. How will it be analyzed?5. What does it mean?
These questions were expanded into an evalua-tion outline. ,
The scientific method is a method of thinkingand we.can apply it in our everyday work.
The -four chapters of this introductory, sectionare designed to exprdin evaluation, its relationi toextension education, how it is used in extensionwork and the scientific approach to the study ofproblems.
In Section II, methods used in the evaluation pro-cess will be discussed in detail.
Reference
1. Rummell, J. F. An Introduction to Research Proceduresin Education. New:York, Harper & Bros., 1958. Chap-ter 1.
THE EVALUATION PROCESS
Chapter VGOALS AND OBJECTIVES
.I. Neil Raudabaugha
In a more comprehensive text we knight be justi-fied in devoting a chapter to- outlining the shadesof difference among concepts such as purpose, ob-jectivd: intention, goal, need, aim, drive, want, andothers. For some ptirposes, we in Extension domake certain distinctions among them. For im-mediate purposes. however. we willtse them inter-changeably, but will tend to use "goal" or "objec-tive" most frequently.
'Need for Objectives
dr How often have you heard renwks like this'?"Everyone knows what we arc tiding to accom-plish; let's get on with the job.' Statements of thiskind are not uncommon among people working withExtension educational programS. Very often theimportance of spending the time necessary to deter-mine and define goals is not realized by laymen,and all too often professionals who work-regularlyin the field of extension education make the sameerror. Because of this. we may find,-professionaleducators. lay leaders and, educational groups soconcerned wit4.their on-going procedures and activ-ities that they 'lose sight of what it is they are tryingto accomplish. They fail to realize that educationalobjectives are the criteria. by which content is out-lined. materials are selectedateaching proceduresand [carping experiences are developed: and pro-gress toward accomplishmeAt. and accomplish-mentare evaluated.
\.Kinds of Objectives
In considering goals or objectives in Extension,it-is essential that we think of them in relation tothe people involved, or to be involved. Some ob-jective; will he group objectives and _some will beindividual or, family objectives. Once goals and ob-
-jectives are determined, they exert a definite influ-ence on the activities and behavior of the peopleparticipating, as well as on the extension Uorkerinvolved. Time spent by extension agents helpingpeople to define their goals and objectives can domuch to help them gain' that needed direction andpurpose which will, in turn, motivate them to in-creased achievements and satisfactions.
Goals and objectives indicate what people aretrying to accomplish; they identify the target to-ward which the action program is oriented. They
[ 18 ]
can be long-time or immediate, they can be generalor specific. In an gducational program it is desir-able tONdistinguish betWeen Jong-time (ultimate)and shorb.run (immediate) objectives. Both havetheir place in extension teaching, but it is importantto the total program that they be compatible.
In any consideration of objectives it, is also essen-tial to distinguish between broad objectives andspecific objectives. The latter are more apt to beachievable, dynamic and capable of doing much tohelp promote action and participation. The formerare achieved-with greater difficulty, largely becauseprogress is not so apparent because of the timespan, and because agreement on feasibility is notso universal.
People working with action educational programssometimes adopt a set of abstract objectives andthen proceed to consider them as though they areshort-run and easily achievable. Such objectivesespecially for extension educational work, fail toprovide needed direction. They limit prospects forachievement and ultimate success of a program, sonecessary to stimulate in he people concerned thedesire t9, strive for growth and development..,.1> pie, both individually and in groups, respondbest Jo specific objectives that they believe areachievable. If educational programs are intendedto stimulate people to participate, dynamic andachievable objectives are requisites. Such objec-tives also serve well as the basis for identifyingevidence of accomplishment for purposes of evalua-tion.
Levels of Objectives
The relationships and differences between objet -.Oyes of extension workers (teachers) and those oftheii. clients (learners) are not always realized. Thisfailure results in much ineffective teaching, andcontributes to the 'development of unfavorableattitudes toward educational programs.
Extension workers, who ;ire not sure what thepeople's 'objectives really are, have to conduct aneducational program.they believe is needed, presentsubject matter they consider appropriate for theprogram, and comfort ..themselves with the hopethey are achieving desired objectives. Actually,they 'may be producing few, if any, changes in thepeople they are attempting"to help. In cases of this
kind, the people themselves. while failing to under-stand and accept, the objectives suggested by theextension worker. may have no idea of their ownobjectives.
When dealing with this frequent conflict inobjectives. it has been found helpful to think ofobjectives On Various levels, each with a place andfunction. Burton. in his book. "The Guidance ofLearning Activities,- has classified objectives intofour levels.:
1. Objectives of SocietyThe general 4,iin of:fiery society is to secure the -good" life f(-4. all ofits members. Surely all of us have spoken glibly atone time or another of high standard of living,maximum production. favorable balance of trade,religious freedom. and such generalities. Remoteand abstract objectives of this kind have a place indesigning all-inclusive ultimate ends, but 'theyraieof little value as objectives for an action orientedextension educational program.
2. Objectives of a Specific Organization or GroupPhis is:,-the level of objectives such as are stated
in the printed materials relative to State and federalactivities of the Cooperative Extension Service. Theobjective toward wh' these agepcies are directedis improvement of the °eon inc. social and spiritualwell being of farm families in general. Objectivesat this level have the laige group rather than indi-viduals or families in mind and; for this reZIF.011, arenot suitahle for designing action extension educa-tional programs below the State level.
3. Objectives of the Extension AgentAn ex-alliple of an objective at this level might be todevelop in livestock farmers an understanding ofthe merits of stilbesterol in livestock feeding. Abroader one at the same level would be to interestparents of junior and senior high school childrenin ways of improving their understanding of teen-agers.
Characteristics of objectives at this level are thatthey consist of what changes agents -desire to bringabout in the people with whom they work. Theyserve as evidence of the ability of the agent totranslate extension objectives into action objeCtives.However, agents who work exclusively with objec-tives appropriate to this level can never be positivehow well their plans fit the needs of their particularclients.
4. Objectives. of .the Clientele These are thethings the people wish to do. or: accomplish. Afarmer may desire to increase his income by tenpercent by improving his cattle feeding operations:a homemaker may want to make a coat-or dress inorder to reduce the social stigma of an inadequatewardrobe and still keep within the family budget.
An extension worker cannot always be aware ofpeople's purposes or objectives. Ile may have to
[19]
set out deliberately to learn what they are. How-ever. an alert worker can spot sonic of them as theyemerge in the course of routine activities. -Ile mustthink through situations and attempt to foreseewhat spec 1 -c desires or needs are likely to reachthe awarewss level of the people. Often it is nec-essary. and desirable, for him to do a little stagesetting and manipulate the environment to help theneeds and purposes emerge to bt recognized.
These fou levels of objectives are not related toeach other in a strictly additive manner. but theycannot be entirely incompatible with each other.The objectives ,,of any teaching-learning situationin Extension must originate in the situation. itself.They should be derived from factors present in thesituation which are of concern to the people.
The immediate objectives of the people are theprogress objectives. These progress objectivesbear upon their inure remote aims. The extensionagent, as teacher and educator, must exercise lead-ership and guidance tosee that a balance in em-phasis between immediate and ultimate objectivesis achieved.
Those most directly concerned often do not real-ize the need for this balance. Agent attempts atdeveloping awareness of this need can he confusing,and have been known to antagonize the clientele.Objectives at any level need to be consistent andcompatible with those at other levels, just asImmediate or short-run objectives need to be com-patible with ultimate or long-time objectives.
Objectives of the extension worker and his clientsneed not be similar in form but they need to havea common core. The people's objectives are thosewhich they believe can be accomplished throughactivities in problem solving projects of their owndesign. The extension worker's objectives are the.desired educational outcomes which he hopes thepeopk;' will achieve.
The extension worker's role is to organize andguide learning experiences so that desired educa-tional results (level 3 objectives) can be achievedwhile the people are achieving their level 4 objec-tives. Failure to recognize the difference betweenthese two levels of objectives has caused muchineffective teaching. It contributes to the forma- .-
tion of detrimental attitudes and to reduced partici-pation by the very people who should be most'con-cerned.
Stating Educational Objectives
Along with an understanding of levels of objec-tives, there is the problem of stating them in a formwhich will be most useful in developing and evalu-ating the educational program designed to dealwith them. This is an important problem for. un-less objectives are stated .so that they are useful.they may be filed away unusekand the educational
4 ";
I program \vill not reflect the effort that went intotheir design. Well stated objectives. have two para-mount uses. The first is to facilitate the selectionand organization of learning experiences; the .sec-ond is to give explicit direction to the -evaluation ofeducational program efforts.
Orfeyather common practice is to state objectivesas activities or tlAings to bQ done which the exten-sion agents, or others responsible for the program,plan to do for people. Examples of this might beto .show how to refinish a piece of furniture, toshow how to useinsulation materials properly, orto present the agricultural adjustment problem.
Such statements indicate what: the person pre-.senting the-program plans to do, but they do notsay who is to be involved or what is to be accom-plished by those involved. Statements in terms ofactivities to be carried on fail to indicate the kindof change or accomplishment that is expected.Activities are not objectives; they are merely themeans designed to accomplish objectives.
Sometimes objectives are stated in the form ofgeneral content areas to be dealt with in an educa-tional program. Statements of this kind do notspecify what is expected of the people involved.For a group concerne(with health. the objectivesmay be stated by listing headings such as: Sanita-tion. vaccination, health insurance. and . the like.Unless they follow in subheadings, such a listingdoes not specify the desired changes tb be madeby the people.
If. as 1.ve stated in Chapter II. edilcation and learn-ing are viewed as processes for changing behavior,then-objectives may be thought of as the desiredchanges. Behavioral changes always occur in rela-tion to some particular situation .aud in referenceto some particular person or group. Statements ofobjectives which give clearest direction to the ex-tension worker are statements which specify thebehavicir desired. thci content in which the behavioris believed to be appr.ipriate. and the people orgroups who are expected to change. The followingobjectives at the extvnsion worker level include thethree elements necessary in statements of educa-tional objectives.
PeopleCor.rned
.
Kind of BehaviorChange Desired Problem Area
Members of thehomemaker clubsCow,time,rs of re-tail, foods in theAtlantic TradeAreaFarmdevelopmentfamiliesFannin; in theLittle SiouxWatershed
and home
To increaseunderstanding ofTo increaseknowledge of
To developinterest in andunderstanding ofTo develop theabilAy to
f
The proposedcommunity centerThe sources offood' marketinginformation
How to makea farm rindhome panEstablishcontour lines
When objectives are stated so that these elementsare present, it is relatively easy to determine whatkind of learning .experiences should be planned,and what the content of these experiences should.be This degree of explicitness in stating objectivesis also adequate to give direction to action forappraising and evaluating results.
The kinds of experiences people need to gainUnderstanding or to develop skillS are different'from those .required to develop mere 'awareness orinterest. Therefore, determination and statement.of objectives are important tasks that require thebroadest possible insight into the processes of edu-cation, the psychology of learning, and the sociologyof groups.
t'e stated earlier in this chapter that objectivesare direction givers for action oriented educationalprograms. and for evaluation of such programs.If objectives are to-serve these two important Pur -,poses, it is desirable to establish standards whichmay be used to judge their usefulness. Here is a'series of questions which an extension worker, acommittee. or any individual or group might use injudging educational objectives. Ideally, a reviewof objectives for an extension educational programshould result in positive answers to all nine)of thefollowing questions.
1. Are the -objectives explicit in specifying thearea in which the changed behavior is tooperate?
2. Are the objectives definitive with respect tokind of behavioral change to be accomplished?
3. Are the objectives stated in terms which iden-tify those who are to be involved?
4. Are the objectives the result of cooperativeaction by extension personnel and others cpn-cerneci to analyze the situation and identifythe problems?
5. Are the objectives compatible with the gen---eral aims of both the Extension Service andthe people concerned?
6. Are the objectives specific-enough to serve asa base for planning, conducting and evaluatingan action educational program?
7. Are the objectives sufficiently limited in num-ber to avoid undue confusion and diffusion ofeffort on the part of extension personnel and/or others involved?
8. Are the objectives achievable, considering thelevel of concern, the maturity of persons in-volved, and the resources available?
9. Are the objectives such that they can relateintimately to both. immediate and long-timeeducational goals, and lead to even higherlevels of achievement?
Steps in Setting Goals and Objectives
Extension workers and those with whom theywork often do not take tithe to define educationalgoals and objectives. 'Sometimes there is resistanceto taking time to define them. This is reflected inthe thinking implied by such statements as, "Wedon't need to waste time doing that," or "Let's getOn with the job:" Another very common feelingabout setting _objectives is expressed this way,"Well, everyone knows what it is we are tryingto do."
One important step to insure the setting of 'ade-quate educational objectives, and their ultimateacceptance, is to provide time and opportunity forthe people who are concerned to participate in theirdetermination. Goals and objectives provide littlemotivation for peOple who donot take part in theirdetermination.
Another very important step is to explore theideas and suggestions of those who identify them-selves with the particular need or problem. Thisprovides an opportunity for interested people toexplore alternatives which might.not have occurredto them as individuals, and to secure some sort ofconsensus. After this is accomplished, statementsneed to, be sharpened up s6,,that they set forthexplicitly the three necessary 4imensions of well-stated objectives, as outlined earlier in this chapter.These statements then can be tested against appro-priate criteria for judging objectives. Once goalsand objectives are set, there should be a final checkto be sure they are acceptable to and understoodsimilarly by all concerned.
Objectives that have been determined and ac-cepted .in this manner meet the criteria for worthyextension educational program objectives and willserve as a firm base for both measurement andinterpretation types of evaluation.
Summary
We start this sectioninto
a discussion of objec-tives and how they fit into the _scheme of extensionteaching and evaluation. By taking thd, time tostate them, we crystallize our thinking on the prob-lem at hand and have abetter idea of how to pro-ceed. In so doing we assemble ott,r forces in, sucha manner that all elements are directed toward acommon end.
We mentioned that there are group and individ-ual objectives, that they can be broad or specific,and that they can be immediate or ultimate. Wealso classified them for use in Extension as objec-.tives of society in general, of the specific organiza-tion or group, of the extension agent, or of thepeople (clientele).
Next we discussed how objectives can be statedso that they are limited in effectiveness by neglect-ing one or more of the requirements of a well statedobjective: Who is concerned ?. What kind of be-havior change is desired? What is the problemarea? We also included nine.test qtiestions to helpdetermine how well stated an objective is in termsof extension educational goals.
References
1. Association for Supervision and Curriculum DeveloP-ment, Guidance in the Curriculum; National EducationAssociation, 1955 Yearbook, Washington, D. C.
2. Brown. I. S. "Working Toward Goals," Adult Leader-ship, Vol. 4, No. 4, p. 4-15. September 1952.
3. Burton, W. H. The Guidance of Learning Activities.New York, D. Appleton-Century Co., 1944.
4. Joint Committee Report on Extension Programs, Poli-cies, and Goals. USDA and Association of Land-GrantColleges and Universities, Washington, Df CC 1948.
5. Lancelot, W. H. Perrne.nRnt Learning. Ncf(v York, JohnWiley and Sons, Inc., 1944.
6. Tyler, R. W. Basic Principles of Curriculum and In-struction. Chicago, University of Chicago Press, 1950.
Chapter VI-
THE PLACE OF TEACHING PLAN''ANALYSIS IN THE EVALUATION
PROCESS
Mrs. Laurel K. Salirosky
We cannot find the worth of (evaluate) anyactivity or Program, or estimate the probable worth'of a new activity or program, unless we know the`educational and organizational objectives of theprogram. It is just as clearly a fact that we cannotevaluate adequately without a clear picture of theprogram that was followed in helping the peopleto reach those objectives, or of a contemplated pro-gram that will be followed. This is a basic andvital step in the evaluation process.
To extension workers; evaluation is determininghow effective their work is, or how suitable *theirplans of . work are for doing effective extensionWork. Therefore, We in Extension must find outif our work has contributed, or.will contribute, tothe accomplishment of desired results., This is the stage in the evaluation procesS where
we study the methods used, or to be used; the sub-ject matter taught; or to be taught;-and the audience,reached. or to be reached. This kind of analysisdoes not answer the westion, "Was I effective?"Instead', it helps answer the question, "Have Iarranged for a plan that has promise of resultingin an effective program?"
At this stage, we determine what the objectivesof the teaching plan were really aimed'at .during thelime the project was being carried. Because exten-sion evaluation is finding the worth of extensionwork, or assessing the potential of. contemplatedextension work or some part of it, we cannot evalu-ate in terms of objectives which. we, as individualextension workers or as part of an extension team,made or will make no attempt to reach. To take an.extreme case, how satisfied would you be withevaluation in terms of objectives. if not one activitythat you carried out could paSsibly have led thepeople in the direction you had wished them to go?
We should also realize that analyzing the possi-bility of reaching an objective through a study ofthe teaching procedures and subject matter willproVide a partial answer to the ultimate effective-ness of our work. By carrying out this preliminarystep in evaluation, either when planning a teachingprocedure or when planning an evaluation project,
[ 22 ]
we determine whether the objectives .were realisticor idealized, material or abstract.
If we find that no attempt was made to reach theobjectives, we can anticipate that the work will beineffective in terms of those objectives. If weconclude that inadequate methods or insufficient.subject matter were used, we can expect theeffectiveness will be impaired proportionately. Ifwe feel that objectives were, properly stated, appro-priate actions were outlined, and adequate subjectmatter was provided, we can anticipate gratifyingresults in the way of desired behavior changes.
Learning Experiences
In educational extension work, the next step afterclarifying objectives. is planning for and definingthe learning experiences to be provided for thelearners. Essentially, learning takes place throughtheexperiences the learner has and the reactions hemakes to his environment. It does not take placethrough what you do as a teacher, but through whatthe student goes through as a learner. There maybe only one teacher in the teaching situation butthere are as many learning experiences as there arelearners participating. s,
In the evaluation process, we assume that%plan-ning for the learning experiences has been carriedout, and proceed to analyze the plan. In order tomake both planning and analysis. more meaningful,we have-some principles for selecting and judgingteaching activities (program execution).. If plan-ning for the learning experiences does not take intoaccount the following principles, its effectivenessmight well be questioned without further examina-tion.
1. The learning experience should be appropriateto the level of education, interests, and facili-ties of the learners...,
2. The learning ekperience should be the con-sidered choice from among the possible waysof accomplishing the desired behavior change.
3. The-learning exptrience should be set up sothat the learners have a chance to practice thedesired kind of behavior.
4. The learning experience should be such thatthe learners will get satisfaction from carry-ing out the desired kind of behavior.
5. The learning experience' should lead, to in-terest in accomplishing other desirable be-havior changes.
!....ct us take an example from extension work. Anobjective in your 4-11 Club program may be to helpboys and girls in 4-H Clubs to acquire the abilityto petfor:n the duties of club officers.. Such anobjective has such long standing approval in exten-sion circles that it requires no evalu on in its ownright. A more appropriate evaluat n for your ownpurposes might be to determine to Nhat degree theclub officers are functioning as the should.
You believe That the club president should knowparliamentary rules and procedures, and be able toenforce them diplomatically. You believe that heshould be poised and confident, and that he shouldspeak clearly and audibly. You believe that thesecretary should keep complete and accurate rec-ords of meetings and .be able to read them clearlyand audibly. You have similar criteria for theother officers.
. Before you can begin looking for evidenco\thatthe officers are functioning properly, you need todo some desk work. Put down and, study the activi-ties you, as an extension worker or as part of the .
extension team, carried out. This should includethe learning experiences you provided for traininglocal leaders to train the 4-H Club 'officers.
Did you arrange for the 4 -H members to be taughthow to do each of the-things that you are expectingof them? Did you see to it that the informationneeded to do these things reached every personwho needed it?. Was it 'in terms that "ley couldunderstand and uFe? What knowled skills, orattitudes ought to' change in these boys and girlsas a result of the teaching activity that took place?By such thought patterns, we move from "Whatbehavior change does the objective imply?" to"What behavior changes could the experiencebring about?"
How did you do your worfcr What did you tryto teach?. Whom did you try to teach? What didyou expect the learners to know (tobe able to do)as a result? Questions such as these fit into thepattern of the preferred way of stating educationalobjectives 'so that they inclucle the three compo-nentS outlined in Chipter V.
Analysis of the Teaching Plan
After you have clarified exactly' what your ob-jectives are, what goal you hope the learners -willreach, what they can be expected to learri or acquireas a result of the teaching or training, and whichones can acquire the knowledge, you have the back-
[ 23 I
ground for an evaluation project on your teachingplan. After these are made clear, the other stepsin the evaluation process are much easier.
Again we want to stress the fact that evaluationincludes not only a measuretherit of the outcomeof your work, but also an analysis of activities con-tributing to the outcome. This analysis-'ought totake place during program planning, or duringexecution; so that urgently needed modificationscan still be affected. If it was not done at eitherof these stages, it must be done as part of theterminal evaluation to eliminate the possibility ofundesirable actions being repeated. Because ofthe importance of evaluation as a .learning experi-ence for extension workers, it is essential that onlythose directly involved in the work do the evalua-tion. ,
We have found that a systematic method of carry-ing out this analysis, following -a definite outline,results in the most satisfaction to the evaluator.The following form has worked out very well. Thecomponents suggested by the headings are essen-tial; the exact form can be adapted to the wishesand needs of the evaluator.
Teaching-level
objectives
Subject-matter
Learningexperiences
(teachingmethods)
Peopleto be
reached
Behaviorchangesexpected.
The order of these headings follows the sequenceappropriate to planning a teaching program, or foranalyzing one already set up. First is the matterof developing and stating the teaching-level objec-tives. The 'teaching-level objectives referred tohere are the extension agent's objectives for the
,given program or unit, as described in the previous,chapter.
Second, for each such objective, certain subject -matter resources are required. List each kind ofappropriate subject matter that was used, or thatyou plan to use,Jafter`each objective.
Third, you list those learning experiences whichyou feel will .enable the learner to acquire thesubject-matter understanding he will need to makethe desired behavior change.. Although it may bepossible for you to list many different learningexperiences after each type of subject matter, listonly those that you used, or plan to use. However,you can keep the others in mind as althrnatives inthe event your evaluation indicates that your,choice was not the best one possible under existingcircumstances, and there is time to make adjust-ments.
Fourth, you list the kind of people who can beexpected to receive the subject-matter infOrmation
through each learning .experience listed. For ex-ample. "those who attend the meeting" are theones t.,-) be listed if some kind of a meeting is thelearning experience. -All farmers in (name of)are;, with an interest, in dairy" might listed ifthe learning experience is reading a news article.(n. listening to a radio bi uadcast deyoted to a topiclike the "weigh-a-day-a-month": plan for improvingmilk production.
Fifth. the expected change of behavior is listed.At this stage.. the most important evaluation of thelearning experience takes place. The kin l of learn-ing experience limits the kind of changes of be-havior that can he expected to take place. Somelearning experiences can be expected to changeattitudes: others can be expected to change skills.Sonic can bring about both kinds of behaviorchange: others may have difficulty accomplishingone. ,
Forexample, it would take a homemaker with avery broad knowledge of sewing to learn the skillof making slip, Covers by listening to a speech onthe subject. :1lore than likely. she would be thesort of person who already possessed the:skill butjust needed to haye her memory refreshed on a.fewpoints. cost of .us would need the added experi-ence of watching a demonstration. hacked up by anopportunity to make one under supervision.
kVhen the above analysis has been completed,you have a basis upon which to decide which learn-ing objectives you are justified in studying. andwhich changes in behaVior you can expect to bringabout. This systematic method of analysis can beapplied with equally beneficial results to a farm.visit, a telephone call, or a club .meeting. It canhe used as an intermediate evaluation of anythingfrom a routine project to your total extension pro-gram.
Relationship Between Judging LearningExperiences and Educational Evaluation
.Judging learning experiences cannot be carriedout without also judging the subject matter usedand 'the kinds of people you expect to reach, as wellas -relating .the learning experiences to the objec-tives in mind. When this broad conception of
judging learning experiences-is employed, it be-' conies a large part of the-total evaluation procedure.
The part that is left to do is to determine if theexpected behavior changes occur, and how satis-factorily.
Stated 'efly, judging learning experienceshelps you decide Nk "1 behavior changes to expect;completing the evaltati .proctess helps you 'deter-mine whiCh expected bel avior changes took place,
tand how well:
[ 24 ]
Summary
This and the succeedi g chapter are devo ed tointermediate types of 'aluation of special Ire inextension teachin: n this 'chapter we have lis-
, cussed hOw ev atio'n enters into judging the prob-able effectiveness of a plan of .work before thereare any results to use in the assessment.
In evaluation of this variety, we are concernedwith assuring a realistic approach to the problemat hand, by stating objectives in a meaningful man-ner; setting, up the action to be carried out, anddeSigning suitable learning experiences. Evalaa-tion is in terms of determining how appropriate themethodology is for accomplishing the objectivesoutlined.
We have.presented a five-part outline for- evaluat-ing an extension plan of work. It includes appraisalof the teaching-level objectives, the subject matterto be used; the learning experiences to be set up,the people to be reached, and the behaVior changesto be looked for. At this stage we have to comparethe elements of the plan as outlined with approvedextension practices. Later, when the plan has beenexecuted, we can evaluate the program in terms .ofobserved behavior changes brought about by theaction.
References
1. llerzog, Elizabeth. "How Much Are They 'Helped?"Children, November-December i958. Government Print-ing Office, Washington, D. C.
2. Tyler, Ralph W. Basic Principles of Curriculum and In-struction. Syllabus for Education 360. Chicago, The..University of Chicago Press, 1950. Chapter V.
Chapter Vii
EVIDENCE OF PROGRESS TOWARDOBJECTIVES
Mrs, 1.(1.,!;w1 K. So rfoNIO
You are well on your way with your evaluationplan after you have selected and defined the objec-tive for evaluation and the work you did to reach it.Thesc, lead you directly into determining the pur-pose of the specific evaluation you wish to carryout, with the necessary clarification and limitingof it.
The next Step is to determine what evidence youwill need to look for so that you will (1) know thesituation, and (2) know that you are going in theright direction, or (3) know that your objective hasbeen reached. This is. true regardless of the eval-.uation procedure you follow'(use a record form,make a home visit: carry out a formal survey. orothers).
Evidence
Our use the word-"evidence" is in accord withdi-Aionary definitions: An outward sign. an indica-tion, that which furnishes any mode of ?roof.
HOw can you tell that you havefFnactGProgress, orthat you have attained the goal toward which yourobjective was aimed? Education is successful onlyto the degree that it has caused changes in the"right" direction. What is the evidence the out-ward sign, indication, proof) of "right" statuschange dr progress?
Evidence:is not always easy to conic by. Ilumanbeings are' complicated creatures and changesoccurring among. them are not always obvious tothe unaided eye. This forces us to look for evidonceof the success of our objectives in terms of observ.able behavior, or results' of/behavior. (e.g.ttearn-Mg how to make a dress: Change in behavior; the'dress itself: Result of behavior.)
BehaviOr, when considered as a result of educa-cation, is, often not in terms of readily observablephysical action. However, we can find evidenceof it by looking fOr improvement in skills. betterti.cderstanding of concepts,. increased 'ability tosolve problemS; 'changed attitudes. appreciation ofdifferent things, shifts in ,values. changed inter'eEts,increased knoWledge, adoption of_ approved prac-tices.
To select the types of behavior to' measure, youmust first .check the' types of change you have been
IF will be trying to help bring about. This involvesreviewing ypur plan of work. Different teachingmethods techniques have different types ofbehavior diaiige as their objectives. The importantthing is to ask yourself constantly, "Was this changesupposed to occur because this particular objectiveis in the teaching plan? Were these methods thekind that could ,bring about successful accomplish-ment of this objective?"
A radio talk be aimed at producing a desiredactiOn or at increr.sig the knowledge of the listen-ems,. A recipe in a iewspaper may be put in forlhopurpose of encouraging people to use food .moreeconomically, tpstefully, or nutritionally. A radio .
talk; news article. or project lesson. may begaimedrat leaching_ why something is important. Onemethod may attempt to bring about a, change inpraCtice; another may attempt to bring' about achange in knowledge, attitude. ceappreciation. Weshall': need to decide which. alid how many types ofbehailor we want to measure,
.Depth of EValuation
I,. In Terms of Changes in Behavior of PeopleOne level at which to measure yTir progress orattainment as a helper, educator, teacher or organ-izer is to find out which changes have then place.
a. In the People TheMselveiHave thay, (thefarmers. homemakers, young peoplehanged theirattitudes or .skills; have they done anything as a -result of the extension activity or Method; has theirstatus changed in any way ,since the extensionaction was staffed?
b. In the ExtensiOn Workers (the' countyagents. local leaders- o officer's) being taught or.trained so that they' can help others learn sonic-thing new
2. In Terns of OpportunityWhen it is difficult'or itripo>sible to leasure progress' at the level oforiginal status or change in people themselves. it isdesirablo to measure works in terms oL the learningsituation w7e have set up. If no learning situationsare set up (no written materials- go out: no talksare given, no demonstrations are put on, no visits
[ 25 1.
..
are made), we cannot expect the people to learn'anything as a result of extension work.
It is logical to assume thlit meetings which have1,000 people in attendance have a greater chanceof Changing behavior than no, meetings at all. How -'ek;er, it would dcspend on the learning situationwhether a ineet.i.i.th 1,000 in attendance, 10,meetings with 100 hi attendance at each, or 100'meetings with le in attendance at each would?pro-vide the best learning opportunity.
In general, the mdre opportunities you provide;and the better they are, the better is your chanceof accomplishing your extension objective. A wellorganized and well attended 4-11 Club event shouldoffer a better opportunity for boys and girls tolearn than a poorly organized and poorly attendedone.
. 13y. appraising the soundness of organization, andthe attendance, coverage and subject matter, youhave some informati ,n about possible.success. You
'still have no real qualitative measure Of educational-change in behavior of the people, Who are reached,but you have a good idea of qUantity.
Levels of Behavior'Change at Which to Measure
There are three convenient levels for doing eval-uative measurements., The first ,one is before anychange, of any further change, occurs (t.11 existingbehavior(positioh before any &tension teachinghasbeen done, or before more teaching is undertaken).This point, level. phase where the learners are be-fore the plan for changing behavior goes into actionis referred to as the benchMark.
The decision whether or not to establish a bench-mark has to be made at this step in the evaluationoutline: If you want to know what progress hasbeeh 'made, or will bemade, you. should have sonicidea of the situation at the benchmark. If you knowpretty well what it is, there is little 'need to estab-lish it formally. It is not necessary if you are
interested mainly in whether the goal is reached.and not in how far learners had to progress to reachthe goal. However, by establishing the benchmark,you provide yourself with a good starting point forprogram .planning by finding out how urgent theneed for change really is.
The second level is during the progress of thelearners toward the ultimate goal of the objective.Sometimes we think of evaluation at this level as aprogress report. In evaluating a long-time objec-tive, progress reports are necessary to tell us howthings are getting along. They serve to tell us ifthe teaching methods are effective, which ones aremost effective and which ones need to be empha-sized in the program of work if the objective is tobe reached, They tell us whith, steps have, beenaccomplished and which remain to be accompliShed.Also, they may. suggest other steps which should be
inserted because of. unexpected developments dur-ing the action.
The third or attainment-of-goal level, is of theclose when.we must determine ,whether or not thefinal goal of the Objective w. reached. When youfind out if few, some, or all of the people reachedthe goal. you _know whether to retain the objectivein the plan of work, redesign it and try again, orsubstitute another ,one. You fan find out whetherteaching methods have been effective or not, andunder what conditions.
'Although it is desirable for most situations tO.check at each level, two of them may be incorporat-ed into a single testing program. -13ecausd it isnecessary in most plans to establish a baSe formeasurement the evaluation plan may call formeasurement at two. levelg, the benchmark leveland either the progress level or the attainment -of-goal level.
Factors for Deciding Which Evidencesof ;Behavior to Use.
Our nett decision, concernswhich changes inpeople are to be considered as evidences of status,progress, or attainment. There are several factorswhich need to be considered in this regard. Mostof them revolve around problems of measurementor Ways to simplify measurement.
' 1. Which Evidences WilNhe Learners Have anOpportunity to Display?Whether or not the learn-ers have the opportunity to develop a new kind ofbehavior, and to display it so that it can be detected,depends on two conditions. The first one is thatthe learning experiences must have occurred. Wediscussed the relationship of the learning experi-ence to behavior change in Chapter VI. The learn-ers must have had a chance to learn somethingbefore it is reason-able to look for evidence thatthey did learn it.
All of the people who had the opportunity, orthose you wish to have the opportunity in the fu-ture, to tharn enough to make a change in' behavior,become the population from which you collect evi-dence. How to sample this total ,population, tomake evaluation procedures manageable, is dis-cussed in Chapter X.
The secourl consideration is whether or not thepeople had a_ chance to display the change of 'be-havior. This means that it is necessary to decideif the people have the facilities, the need, and the'motivation to make the change of behavior a partof their thinking or living habits. Only where afavorable or permissive climate for such changesexists isif meaningful to look for- evidences of teach-ing success.
This factor limits the evidences much more inconnection with yoUth work than it does with adult
[ 26 ]
3 3is
education processes. It is true that favorable learn-ing experiences can beset 01) with equal facility foryouth educational groups (the schools, 1'M«' work,4 II ('lutes( or adult education work. However. afavorable cliMate for displaying evidence of bp-havior change is often, long delayed in connectionwith youth education work.
A. Will the Things Yos, Can Count Be Adequateas Evidence?These outward signs are the easiestresults to use in evaluation. and they make 'nunbetter .working materials for measurement tech-
HoweveT, if you do not consider these,types of evidence as adeqUate proof of status, pro-gress, or goal attainment, you. NviU have to thinkabout tvays of assessing intangible evidences, suchas changes in knowledge, attitude, or thinking.
:3. What Is Your Own Facility in Measuring Dif--ferent Types of Behavior?There is no questionabout the fact that it is easier to measure. changesoftehavior which have tangible evidences for proofthan -it is to measure those which have intangibleevidences. The fact-that a -homemaker has madeuse of a recipe she has seen in a paper is easier todetermine than the fact thiat'she no\C has a betterunderstanding of the value of good nutritioii inchild development. The fact-that a family Othersaround the'kitchen table to plan the dirties of eachmember is not difficult to ascertain. But. thetheror not thelamily now gets along better as a resultof this planning' experience, is an :intangible resultthat is hard to measure.
. .
To determine the realtesults of extension teach-ing, we cannot ignore intangible results Just be-cause they 'are hard to measure: On the other hand.,while we are becoming accustomed to evaluation.procedures, it is 'highly desirable that we become-faMiliar with the. techniques. methods, And pro-cedures by measuring those results which are easierto measure. When the techniques are mastered,we can use them to measure those types of resultswhich are more difficult to measure.
Just as ive learn to read by starting With simplewords, simple sentences. and simple. stories. so welearn evaluation methods by starting with thesimple and advancing to the more difficult measur-ing devices. Keep. in mind. however, when youbegin with the easier methods, that ,_you do not getthe idea that all' evaluation is concerned wily withtangible results that can be seen and .counted.
4., How Will You Limit the Number of Evidencesto Look For ? If the list of evidences is very brief,and to look for all of them would not require a greatdeal of work, all of them should be included in ycurplan. If the objective to be studied has been de-fined and limited so that it requires only one or a
overy few actions r changes in behavior to reach it,
those are the ones, and the only ones you need tolook for.
More often, however. 'there are numerous evi-dences of change of behavior. When such is thecase. there are three frequently used ways ofselecting which ones to use in your evaluation.
a.' Study the list of evidences to see if thereare any readily apparent' types of changes that, byoccurring, would serve as clues to the fact thatother. changes must have been .brought about.,Sometime's only these discriminating evidences, theones which can be taken as indicators of others, areenough to look for. In a progress report, however,be careful that those actions which are selected asindicators of others are adequate to establish statusof steps in progress.
b. If theie are no such discriminating evi-dences, (g- if there are too many of them, study allobservable changes to see which can serve as 'themost imporfant .indicators of change in behavior.If yoii cannot look Tor all changes, certainly thoseyou ,consider mostiinp.,rtant. Will-be of most valueto you. ,P
c. If your 'gong list. cannot be reduced byeither of these methods. yon; chi reduce it arbitrar-ily, by -random selection4.ChONng a random apractical- number of them. areful study. Ran-dom methodology 4j11 be sed in Chapter X.
5. What Will You Do AbOut the Time Factor?Selddm is a Change of behavior, whatever the type,apparent immediately upon recep`ibn of the teach-ing stimulus. For certalh types of change, Such asin attitudes: appreciation, fnowledge, or value. a
change in thought processes may takelace immed-iately. However; the learner will need time tothink over the material, weigh it in his mind, andbe exposed again to his customary sources of guid-ance, before a change any permanence takesplace...
Knowledge which is evidenced immediately afterpresentation of new inforination may be fOrgottenjust as quickly. When the desired change in be-havior is a change of practice, the time ele.ment isextremely important. Some things cannot be clone
, until. '!'or example, a certain season has arrived,certain materials, are _assembled, money, becomes .available to buy what is needed, or the fainily is alltogether. Some things, such as remodeling a house.take a long time to do. Some things take place instagesplanting thlilgar,den, harvesting it, canningthe produce, serving balanced meals throughoutthe year.
In planning the evaluation, therefore, it must bedecided how long it is reasonab e for you to delayyour decision as to whether or n t a certain. practicehas been adopted. Ifidgment ma t be used in thisphase. We do not know exactly how long we should
[ 27 ]
37
wait lifter a pefson has had a chance to carry out apractice. and does not (lo so. before deciding thathe is not. going to do it. The'. longer he refrains,certainly. the less likely it is that he will-ever carryit out.
On the other hand.. there is a limit to the timeWe can wait and still expect to find out exactly whatwas accomplishe,c1:. For daily happenings. 2 or 3days may bracket the memory span. Regardingseasonal activities. we might be justified in expect-ing a respondent to remember an important hap-pening during the rest of the year.
When reporting memory items, it should be made'clear that the data are from memory and not fromrecords. The passage of time dis-torts accuracy ofrecall considerably' in excess of a straight . linerelationship. Periodic: ob -rvations spread over theperiod. rather than a' single terminal observation,will improve accuracy of recall. For sonic typesof informiltion, you may find it best to collect dataat different times. because the various actions ordesired behaViors may be affected differently by
G. What Decisions' Must You Make RegardingUnits?Will the evidence be by degrees of attain-ment. a-Mounts of things accomplished. or lengthof time an action has been carried' out? Will evi-dence of a year-around gar ten be in terms ofcertain vegetables raised. amounts of certain vege-tables canned and stored. or in other terms? Will-it be the average number of,quartS of vegetablesper persona of will it be the per&ntage of thehomemakers who can a recommended average orminimum amount of vegetables?
What will he the evidence you want or need toPorn', a basis for your analysis? What will be theunits? Will they be comparable, or will you haveto convert them before you can make comparisons?
7. What Face Data Will You Need?We thikOf those items Of descriptive or background infor-mation needed from. or about all individuals fromNO10111 we. get our evidence as face data. They in-clude stiO information as years of schooling, age,sex, size of farm. number of children in the family,incoine,and so on. Face data are used for threemajor purpiNes.
a.' Classification of Respondents,Usually itis desirable to find out if there is a difference in thereplies or reactions of individuals possessing differ-ent degrees of ce.rtain characteristics, This requiresan analysis of the replies using the classificationsystem set up for a particular
ifas a
sorting.device. For example, if you want to knowif there is a difference in replies among those whohave had different amounts of formal schooling,you would need AO. include a face data question on
number of ,years- of formal schooling. If part-timefarmers are to be compared with full-time farmers,appropriate face data (tueseions. must be includedto provide that information.
b. Purposes of StandardizationTo comparethe findings from one study with those from an-other. the individuals in both studies must be com-parable, at least in terms of the sort factors underconsideration.. The findings in a study of commer-cial farmers are not directly comparable with find-ings in a study. of sOsistence farmers. If eitheror both studies Nygifno face data questions on sizeor type of farm, the findings from the two cannotbe -standardized lot- pur )oses of studying the rela-tionship of size of farm o ration to other aspectsunder. consideration.
c. Purposes of ComparionFrequently it isdesirable to compare data from kstudy. of 'only apart of the population with data from a study of.;the whole population (universe). Usually censusdata are used for such purposes beCause they dealwith the whole population and because they areabout the best source of depth or trend information.In such cases. face data questions comparable tothose used to secure census data'should be included.
. For example. if the study data are Jo be checkedwith census data OD age, residence, or similar classi-fications. face data questions must be includedwhich will provide data suitable for assembling intothe categories used by the census. When compari-sons are made between studies or with census data,it is vital that data be secured in sufficient detailso that they can be converted to comparable unitsor reporting intervals, although they do not haveto be colle2ted in identical form.
This means that in planning an evaluation. andespecially while constructing questions, the analysiswhich will eventually be made of the results mustbe given serious thought. Think over the teachingmethods and the informational materials. Whatdo you want to know about the behavior changesyou are tryhig to bring about? What human re-actions -do you want to explore?
Do you expect certain types of peoPle to react.more favorably than others to your teaching? Doyou expect that certain mental or material posses-sions or qualifications will make a difference? Ifyou do. be certain to include appropriate face dataso that you can sort on these characteristics, andevaluate their effects properly.
Face data will help you ascertain differences,'make comparisons with other studies, and checkthe probable accuracy of .your data, All of thiscontributes to your major purpose in evaluation.truing your work and making new and better plansfor your work.
[ 28 1_
Summary
In this chapter we have probed more intensively'into the problems of evaluating during the progress
a prGgram. During this dynamic stage we mustlOok fur evidences or results which we can use to:assess the progress of-our work. We might preferto look for them in terms of changes in people, but,
\ because of the delay before most changes actuallyoccur, we might have, to settle for counting ()ppm.-funities provided for changes to occur.
We suggested three stages when evaluation is inorder, referring to them as the benchmark, theprogress report, and the attainment-of-goal levels.and suggested sonic techniques appropriate td each.We then devoted the balance of the chapter to dis-
,[291
-cussing svveirlVtoss t-o consideritrauciding-wrilitevidences to use in evaluation., We gave specialattention to the last factor. the matter of face data,because of its importance to the more detailedtypes of evaluative action we will be discussing inforthcoming chapters in this section.
References
1. Smith, E. It. and Tyler, It. W. Appraising and RecordingStudent Progress. \ ew York, Harper and Brothers,1942. Chapter V.
2, Tyler, 11. W. Basic Principles of Curriculum, and In-struction.- Syllabus for Education. 360. Chicago, TheUniversity of ChicagO Press, 1950. Chapter II, pp. 15-34,
a
Chapter VIII
IDENTIFYING PROBLEM AREASAND STUDY OBJECTIVES
Ward F. Porter
In this chapter we return from our explorationof the various types of evaluation to a brief account-ing ofthe initial phase 4f the evaluation process.This chapter is aimed at directing our thinking tothe "big picture" of evaluation planning, beforemoving on to specific techniques of evaluation.
Our immediate objective is to help the readerdevelop a better understanding of the processxvh6.:,reby appropriate research or evaluation prob-lems are identified and study objectives determin-ed. We hope to emphasize the urgency of doing athorough job'of understanding the problem beforestarting to evaluate if. All too frequently. theimportance of this first step is not properly appre-elated nor sufficiently understood. The net resultit that much needlessly slipshod evaluation is turn-ed out and. unfortunately, gets used withOut ques-tion in-policy making and program planning actions.
The Importance of Problem Identification
Precise identification of the problem needinginvestigation is essential from many standpoints.This step helps the extension worker .clarify histhinking and facilitates the. selection of suitablestudy objectives. It also serves as a basis for sounddecision-making regavling evidence to look for,data to be collected, and analyses to be made. It
, sharpens up his interpretations and suggests waysof getting the results into use.
The need for evaluation and research in Exten-sion grows out of our role as educators and theresulting necessity for determining the effective-ness of our programs and methods. Only in thisway can We make necessary and realistic modifica-tions in future actions to aSsure maximum benefitsfrom our extension teaching. In addition, all ex-tension workers are confronted. with the problemof identifying the needs and interests of theirclientele.
Identifying these needs k.becomes an objective.sufficient importance to demand the most carefulevaluative and research techniques. AlthOugh.some situations are such that relatively informalexploratory devices will serve, others may require
[ 30 ]
full-blown,formal studies. Whatever measures areused,' fhe results will be applied in determiningaction programs, in deciding .-upon appropriatemethods, or in establishing benchmarks by whicheducational progress can, at - a later date, bemeasured.
Selecting a Problem for Study.
The fact that you have identified a problem areadoes net signify that you are ready to begin your,evaluation. More often than not, the problem youhave selected is much too all-inclusive to be handled,adequately in a single evaluation. There is still thematter of identifying the, specific prblem for yourevaluation.
There are many factors that need to be consider-ed before selecting our SpeCific problem area andpresenting it in a form suitable .for Qvaluation. Wemust consider our own personal and professionalneeds, and interests in-the matter, as well as theneeds and interests of our. associates, oui lay lead-ers, and our clientele.' Obviously, there willsome overlapping of interests and .concerns, as wellas occasional conflicts. Any differences, of course,must insome way be resolved:
We must decide not only the general area of con-cern; but also the particular phase or aspect that isto be studied. As an example, let us take the Caseof the extension Worker who would like to knowhow effective he has been in reaching a certaingroup of families with a particular program. Al-ready he has to a degree, limited his area for studyby specifying a broad problem area. Before hehas a study he can cope with; however, he will haveto narrow it down further. He Might .choose toselect acceptance of a 'single practice as an indicatorof how well he has succeeded with his educationalprogram. in short, if his prograrninvolved pasture.improveme)1 he might select just one of the sev-eral recommended practicesperiodic clipping, forexamplefor the- detailed study
'The decision as to which problem is to be investi-gated at any one time inevitably must require somejudgmen't (evaluation) as to the relative importanceof the many different problems. The-,agent mighttry outlining worthwhile areas of study, some prob-/
.7"
leins in each, and' the specific topics which couldbe studied under them. Ile might organize theminto sonic form of priority listing in terms of theirprobable importanceto himself, his associates,
lay leaders or his clientele. Then. by adoptingthe policy of "first things first" .as in programplanning---he can decide on his plan of action.
Such a plan should be executed with care. how-ever. as failure to correctly assess the relative im-portance of the many problems demanding investi-gation can result in frustration and discouragement.This is another way of saying that evaluation, likemost things that are worthaile, requires consider-
able planning.Time is another factor which must be given ade-
quate attention. An all-too-frequent criticism ofevaluation plans. and research in general, is theinadequate amount of time allotted for the kind ofinvestigation that is outlined. Alequate attentionimust also be given to other types of resources whichmust he employed in the study, such as personnel.finances, skills and know-how, equipment and facili-ties, and related items.
Unfortunately. these resource factors are notalways considered carefully enough in planning anevaluationor survey. Often the result of suchoversight 'is a study that never gets finished, some-times because funds are expended before the datacah be analyzed. Another distressingly frequentoaurrence is when researchers, with an impressivearray of electronic equipment at, their disposal,dash into the field with a carelessly constructed,"shotgun type questionnaire which is utterly in-capable of-sectiting valid, reliable, and objectivedata.
One other important preliminary considerationis the amount and kind of information already athand that may be relevant to the problem. Solu-tions, or at least objective information from whichinferences can be drawn, may already be available.In this event, duplication would be a needless wasteof timeand other resources.
One note of caution is in order in this connec-tion. Inferences drawn from other studies or re-search findings of presumed relevance should notbe applied to the new setting without thoroughChecks for comparability. Overgeneralization ofresearch-findings, without regard for comparabilityof 'situations. people involved, research techniquesused, and related factors, is altogether too common,and sometimes fraught with serious consequences.
Selecting and Defining Study Objectives
Once the problem area ,has been identified. itbecomes a matter of pinpointing the specific objec-tives of the forthcoming study. In a sense. theidentification of the problenns) may be thought of
[
as establishing the overall purpose' of the investiga-tion. That is, a study may be undertaken to estab-lish benchmarks for later program or methodevaluation. to indicate needs and interests of theclientele. or to establish an estimate of the effective-ness of a meeting in changing human behavior. Onthe other hand, the proper statement of studyobjectives indicates specifically what aspects of theproblem are to be emphasized.
We can illustrate this distinction between iden-tifying the problem and setting up the objectivesfor study purposes by this example. A homedemonstration agent might have the problem ofevaluating a leader-training meeting. Iler studyobjectives might be as comprehensive as the educa-tional program involved at the meeting. At thesame time, she might quite properly limit herevaluation, because of other-pressing demands onher time and other resources, to a single aspect ofthe program. She might. for example, be contentwith determining the number of leaders attendingthe meeting who learned how to freeze peaches,even though freezing procedures for several otherfoods were discussed,
It is important in this connection to distinguishbetween educational objectives and study objec-tives. Previously, we defined educational objec-tives in terms Of Changes in human behavior te.g.,an increase in knowledge. understanding. or skills,or changes in attitudes or opinions). Study objec-tives may be phrase2d in terms of educationalobjectives, but do not- have to be, depending uponthe nature of the problem.
In a strictly situational type Of analysis, such asa countrywide survey, no teaching objectives assuch need be directly involved. In an evaluationof the effectiveness of an educational effort, how-ever. study- objectives most certainly would haveto be consistent with the program's specific educa-tional objectives. In either case, of course, thescope of the investigation may be limited.
The study objectives also can be stated in termsof learning opportunities provided. Instead ofattempting to appraise knowledge or skillg acquir-ed, the evaluati can be limited to determining thenumber of exposures to the learning situationhow many attended the meeting, how many heardthe radio program, how limb made the field trip,and so on. As indicated earlier, such opportunitylevel evaluations have their shortcomings andlimitations.
Stating Study Objectives
The factors mentioned previously as affectingproblem identification. such as resources available.are all equally relevant in selecting, defining. andamplifying study objectives. Here again, it may
r" tic ile!i;ltil to list the various potential study objcc-encompassed by thc problem area selected,
it) the order of their importance and then considerearn the light of resources available.
.tVe can help ourselves a lot when it comes to0).14.k:tine. i.rialyzing and interpreting ()tn. data ifN',0 lit) a good job of stating our objectives. Themore precisely we state them the easier it will beto design questions that v: ill secure the informationto n7cd.
Otte device for gaining precision in our statement(t1 udy objectives is to develop a series of basicqueLtions around which the investigation can be:focused. In a study of the effectiveness of a mar-kcting-information-for-colit;iimers (MICi program,for example. one of our study objectives might be:To determine the nature and scope of the MIC radioaudience in "X" County.
We could bring this objective into sharper focusby listing several basic questions that we wantansvt ered. Such a list might include questions ofthis hider: i 1 t flow many farm families in "X"County have regular access to a radio that is inworking order? 121 How many of these familiesone or more members oft listen regularly (over
of the broadcasts) to the MIC radio program?I:it What proportion of these families/live on farms?In the open country? In small towns or villages?In cities over 10,(100 inhabitants)?
In the inure formal and comprehensive types ofresearch projects. researchers follow up their state-ment of objectives by working up appropriate hypo-theses relating to them, which they procee' to testby statistical and other methods. These hypothesesare their considered opinions, based on inferencesof what they expect their research will find outregarding their objectives. The hypotheses arethen tested b. the data collected and either sup-ported or rejected, in part or in whole.
The hypothesis becomes the focus of the investi-gation. directing the attention of the researchertoward certain types of facts. By so limiting ordelineating the study, the random and unsystematicsearch for facts is avoided.
c.
[32]
An example of one hypothesis that !night betested in an extension study is the followingOesultdemonstrations are -More, effective than the' massmedia in promoting the adoption of certain recom-mended practices among low-income fat in familiesof "X" County. To confirm or deny this hypothesis,we would have to work up a series of questions tohelp us determine, (1) the degree of effectivenessof the result demonstration. and 121 the effective-ness of the mass media. both separately and collec-tively. First, however, we would have to selectsuitable practices and devise some criterion ofeffectivenessan exposure index. an acceptancescore, or similar device.
Only when the study objectives are preciselystated are we in a position to plan and carry outthe other phases of our evaluation or study in aneffective manner. Time and effort devoted to thisinitial phase of the process will prove very reward-ing in terms of the quality, reliability and useful-ness of our results.
Summary
In this final chapter before elaborating on thetechniques for assembling and analyzing evaluativedata, we have briefly discussed ways and means ofselecting and stating our evaluation problem andstudy Objectives. In essence this is largely a mat-ter of identifying problem areas of major interestand concern, selecting one or more for immediateinvestigation, determining our study objectives, andpinpointing the major questions or hypotheses thatwill serve as the focus of the study. Althoughsimple-sounding in outline, each step merits carefuland detailed consideration. The success of theentire evaluation process will depend on how welleach step is worked out.
References
1. Good. C. V.. and Scates. D, E. Methods of Research,New York, AppletonCentury-Crofts. lnc 1954.
2. Partcn, M. B. Surveys, Pools, and Samples, New York,harper & Bros., 1950.
Chapter IX
SOURCES OF DATA
Ilurciry'gra
As we stated in Chapter VII, it is very commonfor the evaluation process to start oft' with a bench-mark phase. The need to know where you are soyou can determine latt.4 how far you got is funda-mental to evaluation theory. To set up this bench-mark. we must learn what we can about the existingsituation.
If we are old timers on the scene, we can use ourown experiences as a source for much of our setupdata. It might also be possible to call on others tosupply much of the information we need on thebasis of their first hand knowledge of the situation.However, even the most observant person needsinformation of a factual or statistical nature tosupplement his_Orst hand acquaintance with thesetting.
For most of our purposes we will find, upon in-vestigation. that there is 11 great amount of materialalready assembled which will be of use to its, forpurposes of control and comparison if not for'actu-ally solving our problem. By using these sourceswherever henever they can help with ourevaluation we are saving valuable time and 'otherresources for use in later stages otour evaluationor for further evaluative actions.
Primary or Secondary Source
Traditionally, sources of data have been classifiedinto two categories, primary and secondary. Ourmajor interest in this chapter is with the latter kind;assembly. of data from primary sources will be thecentral theme of the remaining chapters of thissection.
We can think of primary sources as the originaldocuments, the first reporting of the facts, the firstgrouping of the raw data. Secondary sources bringtogether facts from primary sources. They canmake a substantial contribution to knowledge byway of analysis and interpretation of primarysources, and by Making information presented inobscure primary sources more readily availableand understandable, but they do not make the orig-inal contact with the unknown. .
Data assembled by means of questionnaires,schedules, interviews and similar instruments aredata from primary sources. records from 4-H
[ 33 ]
members, result demonstrators, farm and homedevelopment cooperators, and periodic reports ofextension personnel are primary sources of data.Summary reports of State and Federal extensionofficials occupy a somewhat uncertain position inthis classification. In one sense they are originaltabulations and compilatibns: in another they arecompilations and interpretations of data fromoriginal sources.
Often we will find it difficult to make the decisionas to whether a source can be called a primary ora Secondary' one. For instance, reports compiledfrom census interview forms are primary sources.Reports compiled from census reports. irrespectiveof whether they were done by census personnel orpeople in other organizations and agencies, mustbe classified, as secondary sources. Many of thereports we in Extension make or help to assemblecontain both original and secondary data. We willsoon discover, however, that we can make good useof either kind by making proper adjustments andallowances,
Although this decision is purely acadeMic in mostsituations, it must be taken into consideration whenevaluation is being planned, It will guide us indeildinx how much trust to place in the datathrough our ideas of the presumed professionalstature of the author. For instance, most of usaccept original research prepared by State andnational governmental agencies and establishedresearch organizations with respect, if not withoutquestion or qualifiCation. At the same time, wewould want to make a personal decision regardingthe qualifications of a committee of club membersor a masters-level graduate student before accept-ing their ?hidings without question.
By the same token, we would want our secondarysource to be at least as qualified as the primarysource it quotes. Much good primary research gets:iebukd during the process of inefficient secondaryanalysis. Similarly, more than one presunkiblysound research organization has committed theerror of attempting to salvage inadequate originalresearch by applying elaborate and unjustifiedstatistical manipulations while synthesizing resultsof the original.study.
There are reasons for preferring a primary
source over a secondary one when both are avail-able. In the case of the former. there are feweropportunities for errors to get into the final manu-script. Counts, sorts, tabulatirAs and interpreta-tions are made by those most to understandthe material. When special definitions are neces-sary, there is greater likelihood that tbcy will beused consistently throughout the manuscript.Groupings will be made in terms of the originalmaterial, rather than the needs of the secondaryresearcher who may never have an opportunity toexamine the original data.
Secondary -SOUrCeS may contain mistakes of .bothtranscription and interpretation resulting from theextra handling inid lack of familiarity with the orig-inal data. and the methods by which they wereassemblitd. Quit'e often there is a tendency togeneralize or extrapolate igo beyond the data with-out adequate jtistification. This is not to suggestthat secondary sources of data have no value, mere-ly that they may have limitations less likely to beencountered in the original source.
Secondary sources may point out important pri-mary sources which we might neser learn of if leftto our own resources. They are helpful in openingup an area and in suggesting directions for individ-ual exploration. They show us where We can locateoriginal sources and what we are likely to find.Their interpretations may help us to understandthe data of the oirginal report. If properly used,they can save us a great amount of time and effortwithout leading us astray from our objectives.
Sometimes data from either source are suffi-ciently comparable so that they can be used toanswer our questions, and further research on ourpart will not be necessary. If either is used inplace of personal research we must be certain tocheck on reliability. accuracy and applicability 'asthoroughly as we can. This is an occasion whenwe can draw on our -own family of expertstheresearch and subject-matter people at the landgrant institutions and in the Federal ExtensionService officesto help its Betide on quality ofsources.
Useful Data Sources for the Extension Worker
No extension worker need fear that he will haveto "go it alone" on a research problem. More thanlikely, he will be overwhelmed by the wealth ofbackground material that can be made available tehim, once he gets tuned in on the proper communi-cation channels.
It is difficult to conceive of a problem that willlie entirely outside the area existing research.Seldom will you find a piece of research whichexactly fits your problem, but you can be assuredthat you will find several close enough to give you
a lot a help in laying out your own study. If you\need help with pulling these leads together, there
are persons qualified to advise you in the StateExtension office, possibly even in the county, andsurely in4he Federal office.
Sources of both published and unpublished mate-rial are too extensive to be given ,comprehensivetreatment here. s However, we will suggest somefavorite stinting points for the exploratory phase ofan evaluation assignment,
General In our work we are likely to think firstof the U. S. Department of Agriculture. The vastresearch resources of the Department are availableto us at little or no cost to us personally. Our con-tact for published materials of the Department isits Office of Information or the information outletfor the apprepriateagency within. the Dpgartment.
We should become familiar with its Bibliographyof Agriculture, published monthly by the depart-.mental library. This contains listings of many, ofthe current agriCultural publications, both foreignand domestic. Incidentally, this is a good institu-tion to become aoqrrEilted with if you arc in aposition to come to Washin'gton, or can make useof its an facilities.
Another good'starting point is the monthly pub-lication of the U. S. Government Printing Office,Monthly List of Government Publications. It has...extensive listings of 'agricultural items\and indicateswhere free copies might be available, or else quotesprices if they must be purchased. In addition,several of the agencies have periodic listings ofmaterials prepared within the agency. Oftenpossible to be placed on their mailing lists toreceive such notices regularly. Yer,i,, often, freecopies are available for distribution, at least for ashort while after publicativn,. from their informa-,tion branch or division.
[ 34 ]
The Department or its agencies assembles a greatmany periodic and special pirrpose reports provid-ing a vast amount of trend. estimating, production,predictive and regulative information on agricul-tural subjects. It maintains yes arch experts. con-sultants and .technicians4 in the ti 'lshington officesand dispursed throughout the St and Terri4tories, and has many others on- foreign as ment.
The resources of the U.. S. Department of Com-merce are also available to us. We can make agreat deal of use of the reports' of its Bureau of theCensus. To learn what this agency has to offer,we should know of its Quarterly List of Census Pub-lications. It lists virtually everything issued bythe bureau and tells you where things are avail-able, together with the price if there is a charge.The U. S. Department a Labor also does researchof interest to us. We can get valuable information
on the agricultural labor force, the migratory laborforce and like information from them.
At the State level, we have the numerous facili-ties of the land-grant institutions (the- ExtensionService, the Experiment Station, the subject- matterdepartments, the agricultural college and experi-ment station libraries) at our disposal. Their pub-lished works generally are available without costto residents of the State from their AgriculturalWailing Room or some central office similaviy titled.
Many of the regional and district people are onlytoo happy to help us. Much service can also berendered by the State Departments of Agriculture.Then, too, there are the larger farm organizations,many of whom have their research and public infor-mation functions well established at both State andnational levels. /
Locally we have the facilities of the County Ex-tension Office, where many of the published mate-rials originating from the above mentioned sourcesare already available. We can call on the localcivic organizations and local chapters of religiousand professional organizations. Then, too, thereare the ether civic officials around the Court Houseand elsewhere'in the towns and cities of the county,as %cell as the elementary and secondary educatorsand education officials of the county.
Home EconomicsMost of the general sourcesalready mentioned will give us leads to materialswith special application to home economics evalua-tion tasks. Howevr, we would also mention sev-eral agencies, or branches and divisions withinagencies of the U. S. Department of Agriculture,whose task is developing knowledge about homeeconomics.
Research in phases of home economics has beencarried on by a succession of agencies within theDepartment since 1894. In the present depart-mental setup, the work is being carried forward bythe Institute of Home Economics in the AgriculturalResearch Service. Special phases of the work areassigned to the Clothing and Housing, the House-hold Economics, and the Human Nutrition ResearchDivisions within the Institute.
Home economists also may be found in Otheragencies whefe, although they may carry any oneof several social scientist labels, will be workingwith the problems of the homemaker. Our owndivision employs home economists in its Federaloffices and the agency maintains a separate divi-sion, the Home Economics Prograi Division, toadvise on and conduct home economics research,among other things. Published works of all thesepersons will be found listed in the sources mention-ed previously.
Other places where we should cast'an interestedglance from time to time, because of the valuable
[ 35 ]
research they do in related areas( (1) The U. S.Department of Health, 'Education and Welfare,where they do so much work related to the home(rural health, child welfare, pure food and drugwork, rural education); (2) The U. S. Departmentof Labor, for its studies of women in industry (andout of the home), and child labor problems; (3) TheU. S. Department of Commerce and its CensusBureau with its many Agriculture, Housing andPopulation reports.
As they do at the national level, most of theorganizations mentioned earlier have home econo-mists doing home economics research at the Statelevel. The land-grant institutions have schools' ordepartments of home economics where they trainhome economists and do 'home economics research.
Most of the counties have home economists onthe county extension staff to head up the homedemonstration work and to participate in the farmand home development work. Most of these peoplereport on research they have done from time totime, and the published reports became available,at least locally, for use of their fellow workers andtheir clientele.
ExtensionWe have already mentioned somesources within Extension but wp would like tosuggest a few more. One source, actually severalsources, of new information on research are thenewsletters of the various extension divisions andbranches to their counterparts at other levels withinthe agency. Most of us at least have access to them,if we do not receive them directly.
One of the special assignments of the Division ofExtension Research and Training is passing alongknowledge of research findings. To this end theyput out a series of periodic reports of use as mate-rial or idea sources.
They prepare annual digests of statistics compil-ed from the annual reports of county and State levelextension workers. Each year they publish Reviewof Extension Research, which is a compilation ofsummary reports of research related to organiza-tion, functioning and administration of the workwhich were completecOuring the previous year.
At five-year intervals they publish. Bibliographyon Extension Research, which is a listing of exten-sion research reports published or reported on dur-ing the previous five-year period. To supplementthese two reports they prepare special listings fromtime to time as conditions warrant. Recent topicsfor speial listings have been radio, television, andcommunications research.
Recently, the Division has begun preparing aseries of popular presentations of results of exten-sion research. This numbered series is called,ER&T Research Summaries. Distribution is limit-ed; presently the Federal staff and the State leaders
of extension studies are receiving all summaries.However, interested persons can get separate sum-maries upon request.
Local Workers will find good source materials intheir office files of annual reports and in theirrecords of farm and home development work andrural development work. 'Similar use can be madeof 4-H record books and result ,monstrationrecords.-/
Summary
If we are beginning researchers (evaluators) weneed not be overly concerned about background,;data for our problem. We just have to know howto get started at digging it out and then direct ourenergies to being selective of what we find so thatwe use only that which is proper and relevant toour work.
In this chapter we have suggested that sources
[361
of data ari: of two varieties, primary and secondary,and have stated a qualified preference for the for-mer. We also stated that many research reportscontain,both kinds of data and that often the classi-fication is difficult to make. We went on to say .
that, rather than make the distinction for classifica-tion reasons only, we should make it for purposesof determining how freely we should use the mate-rial in our own problem's.
We mentioned some favorite ways of gettingstarted on the task of locating background materialfor evaluation work and suggested sources at differ-ent levels Wher.e research results are made known,or where we can direct inquiries about specifictopics. We gave some leads for securing bench-mark and other data for general agricultural exten-sion problems then suggested some for homeeconomics problems. We closed by mentioningsome specific materials which are available fromthe Division of Extension Research and Training.
3
Chapter X
SAMPLING
Mrs. Laurel K. Sa:,rosky
Extension workers usually reach and teach morepeople. than they can hope to get evidence from todetermine whether their clientiele has need for,or has experienced, an educational change. In away this is unfortunate, as the way to get completeand reliable information is to go to all the peopleand find out from them, in one way or another,whether they need or have learned the things youare trying to teach. Since this is seldom possiblein extension work, we have to get information froma part of the people about some of the things taught.
When you choose only part of the people fromwhom to get information, you are sampling. Sincethe methods we use in selecting the sample affectso greatly the reliability of the information weassemble, we need to have a fairly detailed under-standing of how sampling is done.
Even though very fevi of you may ever haveoccasion to develop a omprehensive- samplingplan, you do use sampling methods more often thanyou may realize and regardless of how simple yourdata collection effort. may be.
This means that sound principles of sampling areimportant to all of us in Extension. Proper sainpl-
..s-eng l principles and rocedures should be in themind. of: every extension worker _whenever he isplanning or evaluating his work. In the balance ofthis chapter we will discuss. some approved sampl:ing procedu s which have found a. place in exten-sion work.
General Definitions and Principles
Sampling is selecting a small number of unitsfrom among the whole group concerned in such amanner that they can be used to make estimatesabout the whole group. Too often, a sample isthought- of as consisting of only the first part ofthis definition; " . . selecting a small number ofunits from among the whole group concerned . ".However, to be meaningful in statistics, and in anyevaluative effort, the sample must be drawn insuch a manner that the entire definition applies.It is essential that " . . . in such a manner that theycan be used to make estimates about the wholegroup" enters into plans for drawing the sample.
[ 37 ]
Four steps are involved in sampling, regardlessof how elaborate the actual execution may become,They consist of:
1., Defining the population.2. Choosing,and executing the sampling plan.3. DraWing conclusions based on information
extracted from the sample.4. Inferring : onclusions back to the total popu-
lation.
t1. DEFINE
POPULATION
o4*,
4. INFER BACK CONCLUSIONS
The accompanying diagram illustrates thissequence of events. The first two steps must beinvolved in the planning stage; the other two con-stitute the analysis and interpretation stage. Thesoundness of the sampling plan will determine howfaithfully we can generalize from our sample backto the population.
The population for our purpose is the total unitsto be involved in our evaluation. In extensionteaching the units are generally people (homemak.ers, club members, project participants, potentiallisteners or viewers, farm and home developmentfamilies) but in statistical considerations they canbe virtually anything (trees, fruit flies, guinea pigs,supermarkets, lumps of coal, fish, cans of corn).Thus, our population is that group of people amongwhom we are interested in noting behavior changesresulting from, or likely to result from, our educa-tional and organizational methods.
The educational or organizational objectives willdetermine which people are to be included in thepopulation for a particular study. An objectiveaimed at finding out what farmers learned aboutdairy herd improvement practices from our teach-ing efforts should be evaluated by surveying apopulation consisting of those farmers who have
dairy herds, or expect to have dairy herds,, andwho were exposed to the learning situation.'
An objective dealing with teaching homemakersbett9r child nutrition practices should be evaluatedin terms of a population cc nsisting of, those womenwho have, or expect to have, children to feed.Again, some criterion of exposure must be included.An organizational objective to set up a countyextension council of local 4-H leaders has a poten-tial population of all the local 4-H leaders in thecounty.
The population should be identified at the timethe plan of work, or the organizational plan, is.made up. If it was not done then, it must be'donebefore the sample can be drawn.
The educational or organizational methods usedin attempting to accomplish the objectives usuallylimit the population to a select group within thespecialized population, as well as within the totalpopulation or the area. Although it is desirableto reach all dairy farmers, or all homeMlikers withchildren, or all local 4-H leaders, it is often impos-sible to do so. Only those who are reachable bymeans of the methods used should be included ina population we plan to sample to provide evidencefor evaluation. 4-H Club members make up thepopulation being reached through 4-H Club teach-ing methods. Only the dairy project members arereached through dairy project work.
Before some of you take exception to the last twostatements, we will pause to consider why they werestated so positively. For our'immediate purposes,we have to ignore secondary. exposure. It is always,possible that there is a larger population than Theone which we can define in terms of our objectivesand -those most likely to be directly associated inthe learning experience. This larger populationmay include.friends and neighbors of the definedpopulation; it can include other members Of thefamily and casual passers by.
However, it is so difficult to .draw this largerboundary that we are forced to ignore it for pur-poses of evaluation. If attempted, errors of defini-tion could result in defective analysis and erroneousconclusions. If we feel we must know the identityof is larger population we can set up a separatesun ey or study for that purpose. It is much betterto (Aerate the delayand drain on resources to makesuch a survey than to hazard a guess based on thesketchy data we may have at hand.
In your program planning you decide whom youwould like to reach or teach. By the methods youdecide to use, you identify those you should reach;or teach. .,then, what you do as a teacher or organ-izer, and how you do it, determine- the populationfrom whom you can collect evidence for evaluation.Who is in the population, and where they are, v.determine what methods you must employ to col-
lect the evidence you will need to evaluate yourprogram.
The size of the population will help us decidewhether or not a sample should be used. If it ispractical to contact the entire population, there isno need to resort to a sample, which always intro-duces some element of error, regardless of howwell it is handled. When we deal with the wholepopulation we do not have to concern ourselveswith matters of testing for the statistical signifi-cance of our, results, or the representativeness ofour sample.
This suggests the next term we should discuss,representativeness. This is the essential ingredientin the last half of the definition we suggested atthe start of this chapter. It requires that the samplefrom whom we get our information must be repre-sentative of the whole population. If our sampleis to serve as a re onably accurate estimate of thegroup, Onust po ss all the characteristics likelyto be encounter in the group.--The degree to ,which/it fails t meet this criterion will be thedegree to which is likely to fail as a spokesmanfor the group, and the degree to which our conclu-sions based on the sample data may be in error.
The way you select your sample will determineto a large degree the reliability. of your evaluationor study. A device, methpd, or technique used insampling is reliable only to the degree to which theresults you get by its use approximate true condi-tions. You may be convinced that at least some ofthe people in your county actually were changedby your teaching effort. If your sampling tech-niques are sound, the percentage of people in yoursample that changed practices should be very nearthe true percentage of all the people who changed
/ practices.We can do two things which will contribute to
the accuracy with which our sample approximatestrue 'conditions: (1) We can see that a sample ofappropriate size is drawn; and (2) we can arrangefor appropriate precautions to assure adequaterepresentativeness.
[ 38 ]
Sample Size, -
A common error in selecting samples is to takerefuge in sheer size of the sample, as though samplesize by itself can assure representativeness. Actu,ally, size alone is no pro& of goodness or represent-ativeness. If the sample is not representative tobegin with, size does not help at all. A sample of1,000, improperly selected, can be less ;representa-tive, and can provide less reliable data, than oneof 100, properly selected.
The size of the population is only par'ially,indi-cative of the proper size for the sample The im-portant factor for deciding on size of th, sample is
the presumed homogeniety of the people within thepopulation. It is obvious, then, that describingsample size as a certain percentage or` proportionof the pOpulation is not a realistic clue to adequacyof sample size. In some situations, a 5-percentsample of a small population may be too small; inothers, a 5-percent sample of a much larger popula-tion may be adequate, or even unnecessarily large..
Proper sample size for the particular evaluationor study depends on: (1) Desired level of statisticalaccuracy, (2) costs in material and other resources,(3) homogenietyk of the population, and (4) Ontem-plated intensityllpf analysis.
Whenever a sample of the population rather thanthe whole population is studied, .the results havewhat we call. an "error," "standard error," or"probable ,error." When used as we are using ithere, this "error" does not imply that a mistakehas been made. Rather, it serves as an estimateof the probable limits of difference between anyresult we get and the true result. The larger thesample we use, the smaller this error factor be-comes.
Figure .I.Diagram showing range within which samplepercentage may fall if true percentage from totalpopulation 'is 50by sample size.
In figure 1, the shaded area shows the range_within which a percentage obtained through studyof a sample is most likely to fall if the true percent-age for the population from which the sample isdrawn is 50. If the sample size is 25, and the truepercentage is 50, the percent:,.ge obtained from asample might be anything from 30 to 70. In otherwords, the error might be as great as 20 percentageeoints in either direction from the true figure.
With -a sample of 100, the percentage obtainedfrom a satriple might fail between 40 and 60 (apossible error of 10 percents gg pointsj'. If the.sar,ple is increased to 500, theptrcentage obtainedalmost alw'ays would be expected to fall within 4.5percentage points of the true percentage.
Percent
. 30 or 70
20 or 80
10. or 90If the true percentage is 10 or 50rc. this is .'-\\the range in which sample percentage may fall
or too .1,.,1,1, 1,,,i,,,i1.1 Ies .0 0 0 , 0 0 C. 0 0 0
N g; 2 .7, 2'
SAMPLE SIZE
Figure 2.Diagram shoWing range within which saniplepercentage may fall if true percentage from totalpopulation is either 90 or I0 by sample size.
The nearer the true population percentage comesto the extremes, TOO or 0 percent, the narrowerbecomes the range of probable error in the percent-age obtained from a sample of the population. Ifsample size is 25, and the true percentageis 90, thepercentage obtained from the, sample will mostlikely fall between 78 and 100.
If the true percentage is 10, the percentage ob-tained from a sample will most likely fall between0 and 22. This means that the error might be aslarge as 12 percent. Referring back to figure 1,it will be noted that for the same size sample (25),the possible error is 20 percentage points when thetrue percentage figure is 50.
We can conclude that the nearer the populationpercentage is to 50, the nearer the probable errorrange will be to that,depicted in figure 1; the nearer;the population percentage is to either extreme, thehearer the probabld error range will be to thatdepicted in figure 2. Regardless of the size of thetrue percentage, however, the larger the sampleize is, the smaller will be the probable error.Observe how the range narrows on each of thefigures as larger sample sizes are used.)
This consideration of the probable error factorsuggests that we cannot predict exact numbers orpercentages for our population as a result of ourobservations of a' sample. However, in much ofour work, exact numbers are not vital. We arenot so concerned with the exact number or percent-age of people who have made some desired changein bpfavior as we are in Ikarning whether or not arel ively large or a comAratively small amountor change occurred.
The cost factor includes time and labor and otherresource uses, as well as cash outlay. Since thisfactor looms so conspicuously in any research orevaluation plan, we must work with the smallestsample we can and still feel confident of getting ausably small margin of statistical error This de-mands that we be especially careficl to make our
[ 39 ]
sample as statistically representative as possiblewithin our size limitations.
Regardless of resource limitations, the samplbmust be large enough. to provide'' the opportunityfor all characteristics we desire to analyze to bein the sample, or reprekntativeness «ill be sacri-ficed and the accuracy of our conclusiorisill hesubject to question. If age, education, farmingstatus. nationality, distance. from town, and otherselected factors.are presumed to have bearing onclesifed behavior change, then the sample must beadequate to assure a strong-likelihood that most ofthese characteristics will be %veil enough represent -ed to permit a reasonable amount of subsortingon,each of them. This means we would want mostof thp age groups represented, most of the farmingstatus groups represented, and so on.
.Ftirthermore, we would want reasonable assur-ance that any special groups we desire to examinemore inensivoly will haile a sufficiently large num-ber present so that results..of'our analyses can berelied upon. A category containing less than 30items have such a large probable error factorthat the findings related to it will have veryif any, predictive 1. slue. For most of our work inExtension, a homogeneous group of 100 items willcarry an error factor small enough to allow forreasonably confident applications of the findings.
If we''should desire to sort our category contain-ing 100 items into subcategories (sorting the re-spondents in ,...the selected age group.into severallevel-of-edyation. nationality, or farming statuscategories) we would have to use a larger sampleso that the statistical error for each category is
'within usable limits.Such decisions regarding the amount of subsOrt-
ing ive plan to do must "be made before we drawour sample if we want usable data. We will -findthat simultaneous sorting on more than two char-acteristics usually require a prohibitively largeand expensive sample, unless the population is anexceptionally homogeneous one.. However, .we- canmake tentative analyses using several sorting fact-ors' if we hold .down the number of categories in aclassification' )older membersyounger membersrather than 0-14, 15-19, 20-34, mid so on, agegroups).
likely than, ot to resemble the poptlation from..which it isitseeiected..
Previously, we have been avoiding the use of"selection" in speaking of sampling. Now that webegin using .it, we must make clear that we areusing it in a statistical sense. Actually we do notselect the sample. What ive do is decide on themethod of selection ancl leave4 final selection tothe 'method.
If the method says we -take every fifth name froma list, starting with the second name, that is whatwe do: We don't -start with the first name on thelist, or take,the sixth or fourth name once in a whilejust for variety. We do not substitute or deliber-ately on-ii? a name. .
Therejmay be occasions when such things .arepermitted, but they must be covered in the rulesagreed upon before selecting the Sample. What-ever the justification for their use, they impair. therandomness and representativeness of the sampleand should be-discouraged, tibleSs there are corn.-pensating reasons for shifting from a strictly ran-dom sample to a modified random, or purposive,sample.,
The primary objective of random sampling is toassure -equal opportunities for all units to beincluded. It is not enough to know that certainbiasing factors have not been introduced at thepoint of sample selection;.it is also necessary toknow that all parts of the population have a chanceof being included.
Our aim in drawing a sample that (will serve itspurpose in evaluation is to have one that will repre-sent the population frem which it is drawn, Ac-cording to Earl Houseman of the AgriculturalMarketing Service, "A representative sample is aprobability sample such that the sampling errorsand biases associated with the method of selection,nonresponse, and estimation are known to tte suffi-ciently Io.v to serve a useful purpose.""
Several methods of.dr,f7wing samples-by, randomselection have Jeen used in extension evaluationwork. We 'will discuss several and mention theknown limitations they place on true representa-tiveness, along with the reasons which help justifytheir use.
List SamplingOne method of sampling is toselect a group of people from a list. Through theuse of random selection methods, every person on.the list will have just as good a chance of beingselected as another. The list "must, of course, con-tain names of the total population in which we areinterested. This method should be used whenevera complete list of the total population is available.
There are two favorite ways of selecting a ran-dom sample from a ,list of names. For eithermethod, the names shobld be arrayd in some man-
The Representative Sample
Rand°Vsampling means selecting a sample insuch a way that each item or person in the popula-tion being studied will have ant equal chance to beselected. A sample chosen by random selection ismore likely to have the different charasteristics ofthe population, and in the same proportions, thanany other sample you can select. If a.sample chosenby this method also has adequate size, it is more
[ 40 ]
ner and numbered. It makes' no difference howthey are arrayed before they are numbered, suchas alphabetically, geographically, by mailing ad-dresses, or by clubs:
The first, and easiest, way is to select ev "nth"name from the list. If you need one-fou of thenames on the list, you select every fourth na le;if you need one-twentieth of the names to makeadequate sample, you select every twentieth name.
To calculate the proportion of the names thayou will require (to determine what the "n" w 11be), divide the number in your population by thenumber you desire for your sample. Convert thisto a. fraction with a "1" for the numerator. Ex-ample: There are 7,697 names on the list, A sampleof 350 is desired. You get approximately 22 whenyou. divide 7,697 by 350. Therefore. your fractionis 1/22 and you will need to select every 22nd name.
To determine your starting point, select a num-ber by some random method from 1 through 22.Use this number as the first one in your sample andtake every 22nd number from that point toof the list.
This method meets the criterion of equal chanceas long as the original numbeg is selected in a purelyrandom fashion. We can resort to a table Of ran-dom numbers to deter-mine the first number, or wecan place the numbers on separate' cards or slipsof paper and draw one from a hat.
Sometimes we -can pick this starting number insome way which might violate slightly our repre-sentati'veness ideal, but which will have a 'symbolicimplication for the study. We might use the dayof the month, the month of the yeat, or some otherdate-baseg starting point related to the starting dateof the study. We might use the hour or minute atthe time the starting number was tricked.
Suppose we sit down at 10:45 ib the morningwith our completed list before us and with our .
sampling fraction calculated. Rather than fumblearound with preparing slips for a dfal,ving, wenight get on with our work by arbitrarily selecting'10" as our starting number (45 would be too largeor our purposes). Then the 10th name would behe first in our sample, the 32nd (10 plus 22) would)e the second, and so on. .
Another method of random *lection from a lists using a random numbers tab! These numberslave been assembled into tables,by purely randomneans. At the time they were put up in tables,!ach number had an equal chance of falling in anyoosition in the tables.
To use such a table, you start anywhere on it andake each number which falls within the range ofumbers in your population until you have enough
0,
and include every number less than 7,698 that wecome to until we have 350 numbers. (For moreinformation about using random numbers tables,-,sec reference 9.)
For long lists of names, this method is very timeconsuming. However, ji is a more reliable methodthan selecting every "fith" name if there is likelyto be any cyclic or patterning effect in the originallisting. Conceivably, you could wind up with allmaleS or all females if you were to draw every "nth"name for a 25- or 50-percent sample of a member-ship list of a young couple's home planning club.
Area SamplingMore often than not, you willfind it necessary to work wit.7.1out a complete list ofthe population. In order to sample a populationwhose individuals are not readily identifiable, thesystem called "area sampling" has been devised.In area sampling you choose a random' sample ofgeographic areas instep:' ;`.'ridom sampling ofindividuals. You base on data secur-ed from all the individuals who live within theselected areas, and who qualify according to yourdefinitiOn fOr inclusion in the population.
To set up an area sample a map of the area con-cerned is marked off into segments which have inthem, on the average, a predetermined number ofunits (farms, households, dwelling units). When-ever possible, boundaries that are easily seen whiledriving along in an automobile (township lines, rail-road tracks, highways, streams) should be used tobound the sampling areas. When an area is sothickly settled that it is impossible to use easilyttlentifiable boundaries, aerial photographs whichreveal more minute details of the landscape (fields,farmyards, gardens, orchards)'can be used.
Before starting to mark off sampling areas, youneed to decide the number of units you wish,have in each area. This number may be of dEyconvenient size, depending on situational and studyrequirements (homogeniety of the population, de-sired statistical accuracy of results, contemplateddetail of analysis, convenience in interviewing).
Next, you start in the upper right-hand cornerof the map and count of a number of 'units aroundwhich you can spot identifiable boundaries. If thenumber is over half as large as,-or is lesS than-haltagain as large as, the desired average number, itcan be considered a sampling area. If the numberis much smaller than the desired average, you com-bine it with the next one you outline. If the num-ber is much larger than the desired average, youdivide it through some means other than identi-fiable boundaries.
Move across th map to the left-hand side, mark-or your sample. In our example, with a popula- ing off areas as yo come to them:. Then move backion. of 7,697, we would use the first four columns ...----Across the map to the right-hand edge. Continue
each row, because of our 4-digit population total, this process until the complete map is marked off'
into sampling areas. Next, number these areasconsecutively, again starting in the upper right-hand corner of the map and moving to the left, thentoward the right, an&so on, in a serpentine fashion.When all areas are numbered, you .sample fromthem just like you would from a numbered list ofnames. Sampling can be carried out by eithermethod described under "List Sampling," althoughthe first method is most often used.
In those -parts of the country where the gridsystem of surveying is used in the rural areas, thesection and township lines provide easily identi-fiable boundaries for area sampling segments. Inrural ai eas in other parts of the country, and in themore densely populate& areas, plat maps and aerialphotographs are used to advantage. You will 'findthat the major problem at this point is gettingmaterials which were prepared recently enough sothat you do not have to take time to bring them upto date before you can use them.
An important advantage in using area samplingis that your respondents are concentrated intogroups by the manner of selection. This holds downcosts associated with travel, maintenance, andsupervision of field personnel during the study.
A disadvantage is that, in order to hold the statis-tical error down to a usable point, a larger totalnumber of people will need to be included in thesample. The geographic areas themselves are con-sidered the sampling units_ for determining statis-tical error. For example, if 65 geographic areaswhich include 325 respondents are covered in asurvey, 65 is considered the sample size, and notthe 325.
_Grid or Random Point Sampling Anothermethod of sampling geographic areas is to use a"spot" method. One way is to lay a grid over yourmap and choose the units located at or nearest tothe points where the lines cross: You might alsouse the segments marked off by, the grid as areas,number them, and select from among them byusing a table of random numbers or some morearbitrary way. You will run into a problem ofidentification here,- however, as you will with anysystem which does not make use of conspicuousnatural or man-made landmarks.
Yet another way is to put a number of dots, hitor miss, over a piece of transparent paper and layit over the map. You can use these points as cen-ters of sample areas of predetermine& size, or youcan take the "n" units living closest to this pointto be your respondents.
Eat? t'-these methods can be deficient in repre-sentativetess if the population is ctkely to be spottyin the way in which characteristics are distributed"throughout the area. However, there is a- knownprobability ratio for being included at each stage
[ 42 ]
al the screening process, so they qualify on strictlystatistical grounds.
Group SamplingSampling groups of peopleinstead oqiindividuals can be done by using- thesame principles that are used in area sampling. Inextension files we have all kinds of lists by clubs,both membership and mailing lists. We can usethe mailing lists to classify people into the geo-graphic areas (such as townships) in which theylive. Then we can interview all the people who livein designated areas, instead of the entire member-ship.
If this still leaves us with too large a sample toquestion, we can sample within the areas by re-assembling the lists by areas and drawing a frac-tional sample from each in the manner suggestedearlier for straight list sampling. This is an ex-ample of double-stage sampling.'" Other stagescan be set up, and often are, but they add immense-ly to the complexity of the study and should be helddown as much as possible.
Using the group sampling technique, we canstudy club effectiveness by stu'dying representativeclubs, or club members located in representativeareas of the county or State.- The technique worksequally well for checking on acceptance rates ofrecommended farm practices and for observingmost kinds of behavior change in relation to theextension teaching practices used.
Unfortunately,, this technique increases statistical-error, but it decreases cost. If we have reason tobelieve that differences are as great within groupsas between them or, conversely, t'nt one group isapt to be very much like another in regard to thematter under Consideration, we can use largergroups for sampling units, thus saving even moretime and other resources.
As with area sampling, it costs more when asample of 5 individuals is chosen at random fromeach of 20 groups chosen at random (yielding asample of 100). However, the resulting sample pro-vides more statistically accurate data than does asample of 10 individuals chosen at random fromeach of 10 groups chosen at random (also yieldinga sample of 100).
Stratified SamplingThus far we have describedsampling methods aimed at bringing into the sample"all characteristics of the people in the same propor-tions as in the total population. However, if thesample is small, it is possible that characteristicsbelonging to small groups within the populationmay be omitted. In order to be sure that certainespecially important characteristics get included inthe sample, some method of stratification is recom-mended.
Stratified sampling involves dividing the totalpopulation into several separate populations in
such a way that these populations are different inregard to selected known characteristics (age, farm-ing status, size of city). If the purpose. of using astratified sample is to assure that certain character-istics are included in the sample in proper propor-tion. the same sampling fraction is used for eachstratum (each separate population).
If the purpose of using a stratified sample is toobtain records from ,enough, individuals havingspecified degrees of certain characteristics so thatthese subtypes can be studied separately, a differ-ent 'sampling fraction can be use.1 for each stratum.Whereas one-twentieth of one of the strata may pro-vide an adequate size sample. it may require one-third of another stratum to provide the same sizesample.
This technique is used very frequently in studiesinvolving farm operators. We rea.lize how widelyfarm operations vary in size, and we also realizethat behavior of farm operators on the differentsized holdings is likely to differ as widely. To besure of adequate numbers of 'respondents withoperations in the different size categories, we setup several different sampling fractions. We mighthave to use all the very largest operators in thearea (1/1 fraction). A 1/10 fraction might do foroperators with holdings in the middle range.Among the much more numerous small or subsist-ence operators, a 1/106 fraction might be adequate.
When this method is used, data from the differentstrata cannot be assembled by simple addition forpurposes of obtaining totals for the population.Data from each stratum first must be weightedaccording to sampling fraction tised. In our farmoperator sample, we would have to invert the frac-tion for each stratum, convert it to a whole number,and multiply- by this term, before adding up to getpopulation totals. We would count our large oper-ator data as observed, but we would have to multi-ply Sour middle range operator data by 10 and oursmall operator data by 100 before totalling.
Aidgment Sampling
. The sampling methods we have been describingdepend on objeCtive procedures Which should re-sult in fairly representative samples (everyone inthe population has an equal or known chance ofbeing selected). There are occasions, however,when you might find it better for your purposes tochoose a sample on the basis of what you believe itshould tcontain. In so doing, you hand-pick eachindividual to go into the sample, .deliberately mak-ing 'up what you feel is a sample representative ofthe population from which it is chosen.
Yates, the English statistical authority,_ has thisto say about this method:
. "The ideal which is aimed at in sampling is to
make the sample as representative as possible, sothat measurements or observations on it can betaken as virtually equivalent to similar measure-ments on the whole population. The fact that thisideal is in the mind of the sampler when taking thesample naturally influences his selection if he hasany freedom of choice. Most samplers when select-ing a representative sample will deliberately rejectabnormal material, or -if they feel. that the sampleshould be-representative of the abnormal as Well asthe normal will deliberately balance up the differ-ent categoHes abnormally.
"Unfortunately the sampler's claims to be ableto select 'a representative sample' by personaljudgment are largely unfounded, and his selectionis in fact subject to r arts of biases, psychologicaland physical. To avoiu these biases and to providean estimate of the representativeness of the sample,'i.e., of the 'sampling error', more rigorous processesof selection has been devised."12
Houseman comments: "It is reasonable to expectthat judgment samples will usually be selective inthe direction of what the mind regards as the Moreimportant elements or in the direction of the lessobscure elements. It is also reasonable to expectin general that the variability would be less withinjudgment samples than within the whole popula-tion; if so, this would mean that distributions basedon judgment samples might be seriously distorted. . . Judgment samples naturally look good to the-person who made the selection, but since a samplecannot be appraised by its-looks, judgment samplersare likely to have erroneous, impressions of theaccuracy of their samples."5
Limiting the Population to Study Size
Within certain populations, especially when vari-ability is great, it may be advisable to define thepopulation in such detail that all or nearly all ofthe me bers of the population which have thedesired haracteristics must be included in thesurvey. n such cases, sampling, as defined tn thischapter, is neither necessary nor appropriate. Ifyou shoUld wish to study 14-year.:old, 1st-year 4-HClub members in a county; you would seldom findenough of them to -make up a population largeenough to sample. You would have to use all ofthem.
We must remind ourselves constantly that, as aresult of narrowing the population to study size,we can apply the findings only to the group wehave studied. Comparisons with similar 'groups of14-year-old, 1st -year 4-H Club members in othercounties may be made, but only with due allowancesfor differences in environmental elements of thedifferent locales. We can draw no direct inferen-C,esto another or larger group or population. Anyone
[ 43 ]
7
who .desires to use such data for purposes of coin-_
parison must be made aware of these limitations,and should use them, if at all, only after carefulconsideration and interpretation.
Testing the Sample
\Ve have statistical formulas for determiningwhether or not a sample is of adequate size toaccomplish the objective for which it is drawn. Wealso have checks on representativeness. We canuse the following methods quite effectively inextension evaluation.
If the population to be studied is a class of peoplefor whom information is available from censussources, we can check data from the study againstcomparable census data. Sometimes, when infor%mation requested for census purposes may not berelevant to our particular evaluative effort, we may,ask a few census-type questions, anyway, so that wecan have data to check with census materials toget a line on the representatiVeness of our sample.
Another test for adequacy is to draw from the.population several other samples of approximatelythe same size as the given sample. Then we cancompare the results for the different samples. Un-fortunately, this can become prohibitively expen-sive. A more economical technique with the same-objective is to-divide the sample into two parts,andcompare,the results.
However, even when such are .used, they do notserve as a complete check on representativeness.Of greatest importance is the assurance that ex-,treme care has been taken in following the sampleselection method, in obtaining information fromthe total sample, and in collecting information insuch a way that biases and errors are minimized.
Summary
Most evaluation problems require sampling. insome form. For that reason, we have devoted thischapter in our methodology section to a discussionof what sampling is and hoW it works.
After some basic definitions and a. rundown ofthe four steps involved in a typical sampling.prob-lem; we discussed several ways to san'iple and men-tioned some,of the conveniences and shortcomings
[ 44 1
n
of each. We included some of the more approvedmethods, as well as several which may have to beused when conditions will not permit an entirelyobjective sampling. plan. We stressed, he import-ance of size and representativeness in a sarriple andsuggested some ways to check on each.
We discussed how probability or chance enters.into the rationale of inferring from a sample to apopulation, Without going into the mechanics of,statistical Computation, we commented. on the im-portance of fusing statistical tests for probableerror and reliability.,
References
1. Croxton, F. E. and Cowden, D. J. Applied GeneralStatistics-, New York, Prentice:Hall, -Inc., 1955.
2. Gallup, George. A Guide to Public Opinion. Foils. 2ed., Princeton, N. Y., Princeton University Press, 1948.
3. Garrett, H. E. Statistics in Psychology and Education.New York, Longmans, Green & 04,-,1939.
4. Hartung, M. L. and Ta'euber, C/ Talks by. Report of' University of. Chicago Evaluation 'Workshop, 1946.
U. S. Dept. Agr., Ext. Serv. Stencil No. 781 (5.46), ,Washington,.D. C., 1946,
5. Houseman, E. E. De'signs of Samples for Surveys. 'Agr. Econ. Res., v. 1, No. 1r-January 1949. (Published,by the-Bureau of Agricultural Economics, U. S. Depart-ment of Agriculture.)
6. Houseman, E. E. Talks with Him in Connection withThis Cliapter, -with Earlier Papers on Sampling, andwith Sampling Problems Arising in. Out _Division'sWork.
7. King, A. J. and Jesson, R. J. The Master' Sample of.Agriculture. Jour. of thr: Amer. Statistical Assoc., V.40, March 1945.
8. Pearson, K., edited by. Tracts for Computors. No.XV. "Random Sampling Numbers." Arranged by L.H. C. Tippett. Cambridge University Press, 32 E. 37thStreet. New York 22, N. Y., 1927.
9. Sabrosky, L. K. Instructions for the Use of RandomSample Numbers. 11 S. Fed. Ext. Serv., Mimeog. 206(3.49), Wa'Shington, D., C., 1949. 'as
10. Sabrosky, L. K. Percentage's Obtained from Samplesas Compared with Percentages Obtained from TotalPopulation. U. S. Fed. Ext. Serv., Mimeog. 205 (3-49),Washington, D. C., 1949.
11. Yates, F. Sampling Methods for Censuses and Surveys.New York, Hafner Publishing Co., 1949.
12. Yates, F. Some Examples of Biased Sampling. Ann.Eugenics, 6:202-213, 1935.
.!
Chapter XI
METHODS. OF' COLLECTING DATAGtomp,
Up to thiS point we have been discussing theobjectives. of the evaluation, evidence of the objec-tives. and the persons or population from whom orabout whom we will collect the evidence. In thischapter we will discuss methods for collecting theevidence or data..
The method of collecting the data should neverdetermine what we collect. The most importantpoint is to be sure of exatly what information weneed to meet the requirements of the study. Theperson conducting the evaluation then needs todecide what is the most practical way to collect thedata. 'Ile needs to consider the type of device hewill need to get zi record of evidence._ kle will alsoalso need to consider the conditions under whichhe will have to work, and other demands on histime. budget and other resources.
Methods of Collecting Data
1. Mail questionnaires. .
2. Personal-interview., schedules. usually usedwhile 'interviewing people in their homes.
3. Distributed questionnaires and ,checklistsall kinds..
4. Group interviews.5. Case studies.6. Systematic observation procedures, ..
7. Systematic studies of available records..8. Tape recorded interviews, discussions or
panels.
Considerations Which InfluenceSelection of Method
The objectives of the study.2. The type of device that might be used to' ob-
tain evidence.3. The -respondents from whom you will need
to get thedata:a. Where and' how can they best be reached
at home, through,The Mail, at meetings, orelsewhere?
b. Will they represent a !sample of the tolz..1population which the extension worker hasattempted to reach?
4. The resources you hzive available.a. Your timedemands of other work.
b. The time of others wno will also work onthe study.
c. The time it will require of the respondent,the person who is to angwer.the questionsor give the information.
c!. Money available for supplies, 'stenographicand statistical help, travel, subsistence.
5. The advantage and limitations of each method.
Analysis of Each Method
In selecting the method to use in collecting data,the evaluator needs to analyze each method to seewhether or not it will provide him with valid and,reliable information. Each method has its advan-tages and limitations. These should be taken intoconsideration in making decisions. Let us firsttakb up the mail questionnaire.
The Mail QuestionnaireThis is sent to respond-ents who fill it out aid return it by mail. Beforedeciding to use this method, there are (-Alain ques-tions.' which we need to consider.
I. Can the questions on the objectives of studybe fitted to a mail questionnaire?
2. Can the question$ be stated in a clear andunderstandable fashion without further .ex-planation?
3. What about the people returning them; arecithey representative of the group with whomwe have been working or of the "population"?
4. What about response rates?
Advantages-of the mail questionnaire:I. It is an easy, quick and relatively inexpensive
method of obtaining data, especially whencompared to a personal interview designed tosecure the same information.
2. It can be planned fo reach a wide geographicdistribution of people including some quiteinaccessible people, provided that their ad-dresses 'are known:
3. It is useful in reaching a relatively homogen-eous, fairly well-educated group.
4: It is well adaptethfor reaching specific groupssuch as dairy farmers, poultrymen, or home-makers who belong to organized home demon-stration groups.
5. It can be free of any interviewer bias; but notnecessarily instrument bias.
Limitations of the mail questionnaire:1. The questionnaire must be short.
It is difficult to obtain detailed qualitativeanswers or to know precisely what the re-sponses mean,. ..,
3. There little control over completeness ofanswers; when the eftsporidents don't under,.stand or don't want to say, they may justleave out the answers to .a question.
4. Those who reply may not be typical membersof the list; those who have made changes inpractices or who are especially interested orwho are especially opposed to the ideas pre-sented are the more likely to reply.
5. Questionnaires should be returned fromeveryone in the sample. This usually requiresf011owup. It takes time to write followup let-ters. to make telephone ,calls and to call uprespondents who fail to reply.
Personal interview SchedulesThese schedulesare filled in by an enumerator or repert6r. This!method is used in a more formal type of study inwhich an interviewer calls on a ,sample of people.The interviewers are .usually neighboring countyextension workers, State extension workers, andlay people, such as extension committee members.
Advantages of personal interview method:1. The people who participate have an opportun-
ity to observe and study situations and condi-tions: they talk directly with people and gettheir reactions to practices and programs.
2. The personal interview method usually yieldsa high percentage of returns, as 'most peopleare willing to cooperate.
3. The interviewer has an opportunity to explainquestions to respondent.
4. Complete answers to all questions can usuallyhe obtained. This contributes' to statisticalaccuracy, validity and reliability.
Limitations of the personal interview method:1. Transportation costs and time .required in a
large area may make the personal interviewmethod impracticable. For example, goodenumerators, unless volunteers, can demandvery good salaries. An enumerator might betied up several hours on a single interview.Unless the interviewers are properly trainedand supervised, the data recorded may be in-accurate and incomplete. A few poor enum-erators (.1ti Aas the study.
3. It is usuall claimed that costs per intervieware higher than when mail questionnaires areused. This may not he true if the area to be
covered is not too great. The personal follow-up of the mail questionnaire in order to securerecords needed may run the cost as. high asfor pc..sonal interviews.
Distributed Questionnaires or ChecklistsTheseinstruments are often used interchangeably. Theyare usually handed directly to the respondents in-dividually or in groups. Respondents answer thequestions and hand them back to the person con-ducting the evaluation.
Respondents are asked to record thbir answersto questions by checking or otherwise indicatingtheir selection from a list of statements. Theseare usually distributed:
1. To all persons in a group or in attendance ata meeting, to be filled out at the meeting.
2. To a sample of people who are participants ina group or 'meeting, but to he filled out athome.
3. To leaders who:ask their neighbors or men-hers of the group for information.
These small report forms are used ext-ftsivelyin home demonstration work. They are used tocollect. evidence of progress madeusually prac-tices adoptedfor use in program planning and inwriting the annual report.
Advantages of the distributed questionnaire:1. The questions have a definite relation to rec-
ognized goals,or objectives.2. The cost is small and they are easy to admin-
ister.3. They are, easy to complete.4., They provide a quick N'ray of surveying the
group.5. When 'used before and after a meeting or a
Series of meetings, they can be used as in-terest getters.
6. If.filled out in a meeting, they can be used asa basis for discussion and can indicate thenext steps in program planning. (People in-volved in filling out the forms are also usuallyinvolved in program planning process.)
7. They can also be used as attention holders andindicators of action.
Limitations of the distributed questionnaire:1. The people who fill out these forms are usual-
ly participants in the program and know theobjectives and may be inclined not to becritical of the existing program.
2. These report forms may center attention on"practices adopted" rather than on otherkinds of behavioral changes.
3. Participants in groups in an area are notusually representative of all people in thearea. Therefore, it is better to intercifiek.v a
46
5G
cross section of people in the area if we desireto measure changes made by others than thosein the organized group. In other words:theseindividually distributed questionnaires -tendto measure progress only of those who partici-pate in the program.
4. They are usually on single topics and there-fore cover only a limited field of study.
Group InterviewsThe leader of the study pre-sents a questionnaire to a group of people assembl-ed in one place. Each person in the group is askedto record his or her answers to the question as itis read. To secure an adequate cross section, thosepeople interviewed must be carefully selected soas to secure representative samples of the peopleabout which generalizations are to be made.
Advantages of group interviews: (Advantages. aresimilar to those of the individually distributedquestionnaires.)
1. If the persons in the group are homogeneousand if the persons attending the meeting arerepresentative, data are easily and readilyobtained.
2. There is very little costeasy to administer,very little travel, a small crew can do the job.
3. Speedythe whole interview can be conduct-ed in a very brief time.
4. There will be few refusals.5. The questions can be used as a basis for dis-
cussion and can help toward program plan-ning, as well as to hold or get interest.
Limitations of group interviews: (Limitations aresimilar to those of the individually distributed ques-tionnaires.)
1. The replies may be unconsciously or other-wise influenced by those with strong opinionsveer group bias). 'Since you ,want indepen-dent judgments, discussion. should be diScour..-aged .during the interview.The people attending a particular meeting areseldom representative of the entire group.
3. The questionnaire used for group interviewingis 'usually short and therefore can cover onlya limited field of study.
Case StudiesThese can be studies of one or afew individuals or cases, chronological report ofthe development of a project or activity, or a storyof an individual family. Frequently they are re-ports of concrete events which are vivid, convincingstories of the effects of a program. They may con-stitute important evidence, f properly assembled.
Advantages of case studies:1. They give concreteness and reality to prob-
lems and solutions.2. They present the situation, reveal import-
9.
J 47
5":
ance of human factors in organization, showthe difficulty of problems as they evolve.
3. Often a case study will bring out importantfactors which are assumed to be generalknowledge and for t at reason go unmention-ed by respondents 1, interviewed by othermethods.
4. They may show how decisions are made.5. They show the importance of timing.6. They show the continuity of development and
how the project or activity unfolds.7. They bring out the problems as .they arise,
how these problems were solved, and how themistakes were made.
8. They afford an opportunity to study a processminutely and to discover its strengths andweaknesses.
9. They are effective in serving as springboardsto further thought and considerationa goodteaching device.
10. They show Extension's contribution in help-ing to solve problems.
Limitations of case studies:1. Case studies may give particularized data,
they may emphasize the unique.2. They may provide data on only 1 or ?Aspects
of a problem, and not on all relevant aspects.3. A case- study is only onespse; unwarranted
generalizations may be made.4. They are time consuming and require training
and experience in observing, recording andwriting.
Systematic Observation ProcedureThis rnethAis used during visits to homes or -farms. Observa-.tion of practices which have been recommendedfor the area are observed and recorded in somesystematic manner.
Advantages of systematic observation:1. The worker knows the recommended prac-
tices.2. He can observe to what extent they are being
adopted.3. He can have them listed on an observation
sheet and check them off. This avoids hit ormiss observation.
4. He gets firsthand information.5. He .can obtain permanent record information
at. the same time to supplement his observa-tions.
6. Progress can be shown.
Limitations of systematic observation:1. The observer may use his own valuesattri-
bute importance to what he thinks is signifi-caAt. An objectiVe checi.c. Sheet helps to --re-vent this.
2. It is difficult to separate observation from in-terpretation, /
3. Enumerator as only his intuition to tell himwhat is representative.
4. Costly when much information must be col-lected.
5. Impractical to do for much information.6. Details important for analysis may be omitted.
Systematic Study of Available Records-4-H Clubproject records and farui and home developmentrecords are frequent sources for this kind of analy-sis. An analysis of .records and reports involvesmuch more than recording, tabulating and report-.ing.
Advantages of records analysis:1. Data can be obtained directly from the rec-
ords without going through a field survey orsome form of interviewing.
2. This may take the time of justgOne person.3. They are reliable if the records have been
carefully kept.4 They provide information from-the: past ,which,
could not be collected today.
Limitations of records analysis:1. Records may be carelessly kept and incom-
plete.Records may not be from a represent-alivesample of the population}.
3. Often there is no opportunity to make checkson accuracy of entries.
4. Records may lack comparability and, 'there-fore, be difficult to analyze.
1. Agent alone, or with other agents from homecounty.
2. Agents from another county.3. Agents and lay people.4. Agents, lay people, and State staff personnel.5. Agents, lay people, State staff, and Federal
staff personnel.
, When Will the Information Be Collected ? It is'desirable to decide considerably in advance theexact period that will be covered byllhe studythemonth or season, a specified time after the workhas bee.ii taught or the actiVitv completed. Theevaluation should be planned so as to avoid heavywork seasons of farm people.
When the information is to b&: collected dependsalso on the other jobs that must be done in theevaluation and the time that must be allowed foreach job.' Very early in our planning we shouldwork out a detailed schedule spelling out, (1) jobsto be done, (2) when each job is to be done, and(3) ho is. responsible.
Wher ill .ttle Study Be Made?Step-by-stepdevelopment of plans have been oovarefl-up to thispoint. The method used in collecting data, andfrom what people, will determine where the studyis to be made.
Summary
The Methods of collecting data should not 4ter-mine what to collect. Information or evidenceneeded to meet the requirements of the study isof first importance. The method 'used to obtainevidence depends on the kind of evidence you needand on the resources you have to collect it.
Some methods of collecting,data are: Mail ques-tiorhizies, personal-interview schedules, cbeck.listsof afi kinds, group interviews, case studies, syste-matic observation procedures and studies of avail-able records.
Planning to Carry Out the jEvaluation
We have discussed seven different methods ofcollecting data. Deciding on the data collectingmethod does not end our decision making action,however. We need to decide not only on themethod of collecting data but also who will be re-sponsible for gathering the information, when thework will be clone, and where it will be done.
Who Will Be Involved in Gathering the Informa-tion?This varies with the type of informationwanted and the method used in collecting data. Ifa mail questionnaire is to -be used, only the countyagent may be involved. However, lay people shouldhe involved in -planning the evaluation and when-ever possible should, be used rn collecting informa-tion.
An important value resulting from a study is thetraining and experience gained by participants.,The people who are involved in the developmentanr! execution of a program should be involvedits evaluation. These may be:
[48]
References
1. Barr. A.: Davis, A.: Johnson, P. 0. Educational Researchand Appraisals. New York. J. B. Lippincott_ Co., 1953'.
2. Parten, M. B. Surveys, Polls, and Samples. New York,Harper and Bros., 1950.
3. Riecken. H. W. The Volunteer Work Camp: A Psycho-logical Evaluation. Cambridge, Mass., Wesley Press,1952. < -
4. Sahrosky, L. K. Discussions Evaluation Workshop,April 5-23. 1954, Aguas Buenas. Puerto Rico. USDA.Ext. Ser. mimeograph ER&T 1- (1 -59). Discussion VIIHow You Will Get the Information.
Chapter XII
INTERVIEWING
mrs.frwelt Fesnenden
Thro\ighout the process of developing a studyplan we have to- keep in mind how the 'informationwill be collected. In this chapter we will discussinterviewing as a. method of collecting data. Agood interviewing job cannot salvage a poorly -damsigned study but poor interviewing can lessen thevalue of a well-planned study.
' We need to know 'what kind of interviewers wewill have before we can decide how involved tomake our plans for collecting data.
.
We must know whether interviewers will, servewithout pay or will expect to be paid, and howmuch, before we can make estimates of costs forthe study. We must know how many we will usebefore wean estlikte low long the data collecting.phase is likely to take, and whether or not we willbe able to handle supervisory details without assist-ance.
An interview is asontact by an investigator witha person for the purpose of securing specific infer-mation. It can take 'place in a face-to-face settingwith the interviewer asking questionS and recordingthe answers he receives from the respondent. Itcan occur in a modified- race-to-face setting wherethe interviewer asks for information from a groupof. people. With thiswill record their answerinterviewer will be coneHying statements and stnrespOiiSer I'.
-A third type-of contactphone, wherein the interfrom ,the respondent via tthemon some sort of tall
Most of us in Extensioninterviewing at one ormot e of thit is so casual that we do' not thinkviewing, and may give no serious thouating the answers. Occasionally, Though, we spot-
A- check some phase of 'our work by interviewingmumbers of our clientele, and in a while weenter upon full, blowi evaluation problems requir-ing organized intervi wingplans.
In addition to supply ng us with the informationwe need for our work, personal -interviewing givesus a better understanding of the people with whomwe work and of the community in which ,our work,
rrangement, respondentsin some. manner and the
rned, primarily, with clar-iving for completeness of
be
is by means of the tele-viewer solicits 'answerse.telephone and recordssheet or schedule.
a good amount ofe levels. Often
f it as inter-ht .to evalti-
[491
lies. This is most true of the more personalizedforms, but is proportionately so for all forms ofinterviewing, All too frequently, however, we donot give much thought about how to do it properly,or to recognize all the occasions when interviewingmay be used to advantage.
Major Kinds of Interviews
Proper collection of data by means of interviewsis a very important tool in research.
As we discovered while discussing other tech-niques used in evaluation, those we can use furinterviewing also are varied, and equally subjectto complicating factors and limitations. We willtalk about the three major kinds of interviewing.,that can be used to advantage in Extension andmention some of their advantages and disadvan-tages.
Face-to-Face InterviewThis is the most per-sonal level of interviewing and, because of the one-intervietar-Dne-respondent pattern, it is the mostexpensive.. It is more time consuhling to make theindividual lime visits. Although there are occa-sionS when it is possible to get the respondent toconic to _your office to be interviewed, most of thetime you have to go to him.
Original transportation costs are greater per re-spondent, and several trips may be necessary, beforethe prospective respondent is contacted and the'interview is secured. There is also th:. strain ofrepeating the processes of gaining admittance andestablishing rapport (making the respondent wantto cooperate). .
To compensate for the heavy drain on resourcesreqtiired by personal interviews, there are severalconvincing advantages; The interviewer may beable to keep his respondent interested and atten-tive for a longer period of time by changing thepace or taking a conversational break whenever heobserves signs of declining enthusiasm. This isdifficult to do in r group situation or during a tele-phone interview.
A personal, visit provides the interychance to observe personal reactionMuch more enlightening than the v
53
ewe, with awhich ,.!an bealized answer
alone. ?le is in a better position to note resistanceand May be able to discover \reasons and deal withthem mere easily in a personal setting.
Even though some people' may respond morereadily to a telephone interview,- the interviewerhas more opportunities to establish friendly rela-tions during a face-to-face visit: A smile, -a com-ment on something visible, a personalized word ofpraise, an expectant attitude, all can be used by analert interviewer to draw out an answer from areluctant respondent.
Telephone Inter ,lewThe telephone interviewmay be very useful for certain .purposes. If theinterview schedule is relatively brief, we can assem-ble complete and accurate information in thismanner. However, it is not so effective when"thought" questions are employed or when datawhich maKnot be readily.available are required. Inexceptional cases, and when skilled workers are:)perating, it is possible. to keep a respondent inter-ested and attentive fora half hour or more.
If the respondent happens to 'be a member of aclub, or some other group which is sponsoring thestudy, and already is familiar with the purposesfor which the daja are being collected, she mayrespond quite satisfactorily to a telephone inter-view. In situations such as this, when good rapportpreviously has been, established, telephone- inter-viewing can be quite effective, and the time and
.money savings become importchrt inducements-.todecide on this method. In general, though, tele-phone interviews are not so desirable for long,tedious ,interviews, regardless Of how vi'ar- pporthas been established. Then; too; the valuek ecru-ing from personal observations are lost, an it isnot so easy to discover and overcome. >' ,a1 reasonsfor reluctance in answering.
Group Interview:This techiiique is not tc.: be; confused,,'ith `rout; counseling or group therapy,4,!ertainly' no: with group discussion. Essentially,it is A device for shortening the data-collectingphase to a minimum: In this procedure, the inter-viewer ,zssembles his prospective respondents in',Ua single group or series of groups, and explains thepurposes of the study to them.
He gives necessary instruc ons and explainsquestions which may not be c ar. Then the indi-vidm. s fill out the form. ev may be asked toanswer each question sim ltaneously, with the in-.terviewer commenting on each as they come to it,or -;.,cri respondent can oil: at his own spee andturn in his schedule when he has worked th oughit to his satisfaction.
in addition to setting up the interview and mak-ing initial explanations tl.k: interviewer is there toprovide whatever help he can in assure c:omplete-and usable record forms. If he is interested in
[ 50 ]
60
,r)
having all of the. respondents deal with the samequestion at the same time, he should arrange forassistants to circulate among the group to aid thosewho are having difficulty with the meaning of ques-tions and to encourage them to answer all ques-tions. The interviewer should discourage conver-sation which might lead to pooling of answers.which means getting one set of answers repeatedseveral times rather_ than several different setsof answers.
The obvious advantage of .the method lies in thesavings of time and transportation expenses. Then,too, the same preliminary explanatory remarks willserve for the entire group. Some of the more ob-vious trouble spots also can be cleared up collec-tively. 44A third factor is that there should be littleor no difficulty connected with gaining admittance,establishing rapport.< dealing with refusals.
If the meeting is called for the specific purposeof filling out the schedule, or if the members knowthe interview is to be an item on the agenda, theywill come knowing the purpose of the meeting and
prepared to cooperate. Similar reactions maybe expected from a selected sample which hasagreed to assemble for purposes of supplying theinformation you need.
41Limitations of the effectiveness of group inter-
viewing are related, (1) to reluctance of respondentsto ask about points which.are not clear to them, (2)to their overwillingness to skip the difficult (verylikely the most important) questions, and (3) to thepooling problem already mentioned. There is lessopportunity to probe for reasons for "yes-no". or"agree-disagree" answers, so requests for free retsponse or essay type ansv. ers are not productive.
Responsibility of the Interviewer
The interviewer has a decidedly important partin the successful comRletion of a piece of research.On him falli; the responsibility for actually contact-ing the respondents and collecting the data. Agood device for collecting evidence' can be madeworthless in the hands of an inefficient or irrespon-sible interviewer. Because his part is so vital inany research project requiring interviewers,. hisselection and training are important steps in theresearch process.
Before interviewers are selected you should havea clear picture of what their duties will be and whatwill be expected of them. The interviewers shouldalso understand their responsibilities and be will-ing to accept them. This knowledge will help yourule out obvious misfits, and suitable trainingshould make the others able to carry out theirassignments in adequate fashion. We can think ofseveral personal and professional responsibilitiesof our interviewers. We will discuss each briefly.
1. 'Maintain a neat personal appearanceAs yourinterviewers represent both you and the age\licywhich is sponsoring you. it-is best that they serVas good representatives. For this reason. yob\should try to get pleasant appearing. neat persons \to serve as interviewers. They stiould-dress appro-priately for the season and -the environment inwhich they will be working. They should be cleanand not given to offensive habits or mannerisms.
2. Locate the respondent, get the facts, and re-cord them The interviewer must be familiarenough with the area to which he is assigned tolocate the proper respondents -according to thesampling .plan for the study, or he must learn howto use the map of the area. If a.person to be inter-viewed is not -available, he should report the factto his supervisor. Under ho circumstances shouldtie make substitutions without authorization.
A. prospective' respondent is not dropped fromthe study or substituted for merely because he isnot at home at the time of the initial'call. If it is(yerrnineA that he will be away for the durationof the fiele. work period, that he is ill, or that hewill not be available for some other reason, it maybe necessary to list him as "not available" or to
Iihstitute for him according to the prearrangedplan. 'Several calls may he necessary to locate aperson or to complete an interview with him if heworks, away from home or is busy.
An interviewer must make every effort to obtainadequate and objective answers to all questions.Incomplete schedules and partial answers tend todestroy she randomness of a sample and make study-findings less useful. The interviewer should havea persona. desire to make every contact fruitful.Notations should be legible, and as complete asnecessary for others. to understand. Researcherscan help interviewers by doing a good job of orient-ing them to the study and by providing them withgood instruments (questionnaires, schedules).
3. Be considerate, and hOnest with respondentsBecause your interviel,vet must be able to think onhis feet and be able to adjust to the situation withlittle or no foreknowledge of what it may be, it isdesirable that he be an intelligent and understand-.ing person: He must deal in an honest and forth-right manner with all respondents, .but he mustknow when to say gracefully that he cannot answera question. However, questions about the purposesof the study should be .answered frankly.
4. Understand the purposes of the studyThegood interviewer realizes that to do his job rightand to have a personal feeling of satisfaction indoing it, he must know what the study is about.what its objectives are, and why the particularquestions are being asked. When he understandthese points thoroughly, he finds it easier to gain
[51j
an inner conviction that the study is worthwhile,and also finds it easier to transmit that feeling tohis associates and respondents. Similarly, he finds,that his work becomes more effective and moreenjoyable.
5. Be thoroughly familiar with the instrumentsIt is very important that an interviewer study andunderstand very thoroughly the forms upon whichanswers are to be recorded. Once he has gainedpersonal- understanding, he can transpose the tech-nical language of the project outline into, easilyunderstood and meaningful language for his re-spondents.
He can gain the desired degree of familiarity bycareful attention during the training phase. and byasking questions to clear up points which fie doesnot understand completely. When he knows the"why's" as well as the "what's" of his assignment,he can do a better job of anticipating questions ofrespondents and of having answers ready for them.
6. Follow sampling instructionsThe interview-er must realize that considerable thought is givento identifying the people he is to interview, andthat he has no authority to take liberties with thesampling Plan or instructions for its use. ,Once heis assigned certain persons to contact, it becomeshis duty to locate those persons and to do all thatis in his power to secure the information.
If there are goad reasons why he cannot carryout the plan, he should report the fact, and explainwhy, so that proper corrective measures may- beapplied. He must realize that he can make thesituation worse instead of better by acting on hisown initiative rather than checking with his super-visor for further instructions.
7. Ask questions eicactly as writtenThe .nter-viewer must realize the difficucity -which lies in de-signing "a question which ;neans the same thing to'everyone. At the same time he must realize isnot wise to modify questions at any time. Even ifthe respondent appears to grasp a different mean-ing than the one he believes a questions contains,he is not at liberty to interpret the question to getwhat he considers to be the proper answer.
Although the interviewer is encouraged to urgethe respondent to think-and to answer completely,he is not authorized to direct the thinking of therespondent in any way. He nullifies the purposeof the study if he rephrases questions, suggeststhings to consider, asks leading follow-up questions,or does anything but encourage the respondent tothink for himself.
8. Check' work fort-completepessrecord It rm should be turned in by 14:. vieweruntil,he has checked it thoroughly for c(;:ilidetenessand accut-acy. It is most effective when done im-
. mediately after the'interview, either before leaving1-the respondent or before going on to the next one.
Editing should be very thoyough at this stage.If the respondent seems to be anxious to have
the interviewer leave, the interviewer may thankhim warmly for'his cooperation and make his checkiii the car. However, he is in a better position to
`make corrections and completions if lie makes thecheck before leaving. Ordinarily, :I remark, like"You have been most cooperative. Do you mindif I glance over this form to see that I have evey-thing?" take the pressure off the respondentand make him .happy.to let the interviewer readthrough the -form before leaving. It may evenprompt him to volunteer. ni(ve useful information.
Se:-,cting'and Training Interviewers
Extension research usually is conducted underthe leadership of extension personnel. For theMost part, data collecting is done by extensionworkers and lay leaderAbSometimes, experimentstation and other research persons from land-grantinstitutions are involved. It is rarely the case thatprofessional research organizations are employedto do research for Extension.
Suggestions that follow will, for the Most part,be directed toward the untrained interviewer. Morethan likely, these will be county level extension per-sonnel or local leaders. Extension personnel donot receive extra pay for such tasks and local lead-ers-are ordinarily not paid for their services.
The absence of direct economic motivation sug-gests two possible limitations on the effectivenessof interviewing by such persons. For one thing,strict attendance and punctuality cannot be -de-manded of interviewers and, for another, theremay be less -feeling of responsibility on the part ofvolunteers.- However, we in Extension have beenvery successful on many occasions in the use ofvolunteer lay leaders and 'county extension person-nel in our research teams.
Interviewing is best done by 'people who are in-terested in doing the job, and are willing to main-tain an objective approach. Before agreeing tohelp With a study, anyone offering his services .asan interviewer should feel reasonably certain thathe will have enough free time to stay with the jobuntil it is completed.
You should mention the probable duration of thework when you make inquiries about interviewers.You should includt. time for training and otherorientation activities, as well as time for doing theinterviews. You should allow no one to participatein data collection for the study who does not par-ticipate in the training phase, anti who is not will-ing to assume all proper responsibilities of inter--viewers.
For most studies, you can work lay leaders intoyour interviewing plans with little difficulty. 'Youhave no housing problem with them and the transportation problem is minimized. They ._probablyknow the area quite well and know how to getaround in it.
They are more likely to have a personal interest,in the outcome of the -tudY and may be in a position to capitalize" on the findings. If properlycautioned, they can be counted- on to get accurateinformation.
9Their familiarity with the local scene
can be used to advantage in identifying and locat-ing respondentS.
Regardless of ,experience, all interviewers re--aquire a certain mount of training for a specific
tresearclf- undertaking. The inexperienced inter-viewer will require more training in the generalphilosophy and., discipline of interviewing. but allof them will require orientation for the particularresearch probletn, and experience in dealing withthe specific researceinstruments which will beused.
We recommend that all interviewers be giventraining in six areas, with the intensity'of the traih-ing-in each area to Liepend on the experience levelof the interviewers for the particular study. Thesuggested areas are: (1) Background and purposesof the .study, (2) sampling ,procedures and how toapply them, (3) interviewing procedures and tech-niques to be used, (4) familiarization with the re-search instruments to be used, (5) experience infilling in and editing Ow record ferm. ank, (6)experience in asking the questions. An .actualfield test to follow the tra....ing session is. veryeffective in !linking the transition from theory toreality.
Almost invariably the length of time that can bedeyoted to training the research' team is limited.
'This necessitates careful planning and efficient useof time. , A minimum-of six hours is recommendedfor larger studies, although a 1(inger period, is de-sirable:
It is a good idea to divide the training-irilo twoparts: (1) Orientation (purposes, background andtechniques); and'-(2) practice (experience in fillingin and editing the record forms and in asking thequestions). We have used the following outline onseveral occasions in a ono -day training session forsome quite'substantial studies. It requires at leastsix hours but might take more, depending on theamount of discussion and elaboration you permit.
(:127
Agenda for Training Interviewers
1. Why we are here. (Resident agent in charge.)a. Greetings mid introductions.b. Purpose of meeting.
-Explanation of...plans for the clay.
ci. Why we wanted to make ,this study; howwe plan to do it. ( I) Review of objectives.
2. What an interviewer does. (Training leadertakes Over.)a. Importance of the interviewer in a research
team. (1) Inportance to success of iiroject.b. Definition of terms. (Respondent, sample.
schedule, refaal. call-baCk. unit, and other'general terms. Also any special terms usedin the particular study.)
c. The sampling plan: What it is and how tofollow it.
d. Detailed explanation of instruments. (Ques-4,tion by question and with opportunity for_ ,
. questions from interviewers for clarifica-tion.)
e. General instructions for 'interviewing.( HaVe a set of instructions to distribute tointerviewers but discuss them in detail, atthis point. Ask. for questions. on any pointwhich k no,t clear to alt.)
f. Specific instructions for asking. questions.,(After a general statement regarding tech-niques 'bf asking questions, use the ques-tions in- the study instrument to denion-'strate how to ask them properly. Alsodemonstrate suitable forms of approachingthe respondent -and of terminating theinterview.)
. ,3. Filling in and editing schedules. (Have inter-
viewers pair off and ;take turns asking thequestions, and answering Thecn. Each thenedits- the schedule he has taken.)
4.* Editing by trainers. (Trainers check sched-ules turned in during practice and point outerrors to ,the 'group to alert them for similarsituations. in their own interviews. Trainersexplain how to avoid errors and how to goabout correcting them.)
5..;Explaining special procedures for the study.a. Procedures for identifying respondent,
making contact, arranging. for interview;dealing with refusals., not-at-homes, call-backs.
b. Where and when to bring completed sched-ules.
c.,,,How to contact leader of survey party forfurther instructions or help in'dealing withunexpected or unusual situations.
6. Making first assignments for interviewers.7. Setting time for general meeting at end of
ft:A clay of interviewing to discuss problems,clear lip questions, and check on quality ofwork.
8. Suinary of day's activities; final request forquestions:
9..'Ciosing words of encouragement, assuranceand morale. boosting.
Interviewing Procedures and Techniques \There are four main stages to making and corn-
pletingian interview.1. Gaining entrance, making the approach and
establishing rapport.2. Securing and recording information.3. Closing the interview.4. Editing.
Establishing RapportA good interview dependsto a great extent on a quickly established feeling Oftrust and confidence in the interviewer by the re-spondent. Sincere friendliness on the part of theinterviewer is a firm step in the direction ofachieving this objective. Techniques for gainingadmittance to a respondent's home and for estab-lishing friendly relations (rapport) vary with thesituation. A good interviewer will learn how tomake on-the-spot adaptations to fit most situationslikely-to confront him.
Some techniques may be acquired from study ofliterature on the subject, but practice is also im-portant. An interviewer needs to try to under-stand'. the reasons for- resistance and to think ofways. to overcome them. The most common rea-sons for resistance are shyness, fear of being soldsomething, feeling of inability to answer questions,and reluctance to talk about personal matters.
Neat personal appearance and appropriate dressare Important in creating good first impressions.Equally important is the general tone of your intro-ductory remarks. They should (1) identify you asthe interviewer, (2) explain briefly the reason foryour call, and (3) ask for the respondent's cooper-,ation.
One interviewer has used successfully the, follow-ing as an opening sentence: "Good morning (goodafternoon, hello). I am (name)." Pause. "May Icome in andvisit with you for a' few minutes?" Thisprovides an opportunity for the respondent to givehis or her name, if it is not already known to theintervieWer. --
Most people invite the interviewer in on this re-Mark. However, if there is hesitancy or if somequestion is asked as to the reason why, the next stepis to explain In a little more detail what you aredai and why. It' may help to include some state-ment let the respondent know that cooperationis impoitant. "You are one of a few people. select -,ed and your opinion (cooperation) is very 'impor-tant," is helpful. .
"How was I (My name) selected?" is a questionoften asked by a person being interviewed. , Theanswer: to this should be honest and brief. Some-thing like, "Well, We .,,,,t visit everyone, so we aretalking to every fifth person on your block (in thisneighborhood, on our list, and so on)," usually will
[53]
satisfy the r- espondent. Sometimes assurance isneeded that names will not be used in the report,or that the. interviewer is not selling a product.
An alert interviewer= usually can pick up some-thing of common- interest to talk about if a personshows signs of wanting t8 cut off the .conversation.If, a homemaker says she is busy, you can recognizethe fact, express understanding, and make an ap-pointment for a return call at a more convenienttime.
Refusals should not be accepted except as a lastresort. If an interviewer liaS a high number ofrefusals, he may need some help -with his tech-niques. On the ether hand, it might be in the bestinterests of the study to relieve him of his inter-
, viewing assignments. Sometimes, the person whosesensitive nature makes him a poor interviewer canmake a valuable contribution to the study as adiscriminating checker or, later, as .a clerical aidin tabulating the data.
Almost anyone can be interviewed if you use theright approach. Do not force the interview on aperson, and never become impatient or angry.,, Ifhe refuses, try to make an appointment for someother time. If this also fails, thank him for con-sidering your request and leave, showing no resent-ment. This leaves the way clear for another inter-viewerthe crew leader, an older (or younger)interviewer, or one of the opposite sexto make atry at getting the-interviei.V.
'Quite often, the respondent will give an inte -
view when another interviewer approaches him.possibly because he now realizes the importance ofthe study or, maybe, because he regretted his orig-inal refusal but didn't want to back down.
Refusals happen very seldorrrso do not anticipatethem. Try to avoid them -but, if you can't, acceptthem graciously. Then forget it. Remember, therewill be some refusals in any survey, so don't take itpersonally if you get one.
Before giving up on assignments, make at leasttwo call-backs where the respondent was not athome on your first call, or was too busy to be inter-viewed. Be sure to keep *a record of incompleteinterviews, explaining what you did, and why youhad to give up on them.
Securing Information Thorough familiaritywith questions helps to make a smooth interview.Under no circumstances r7ay interviewers changethe wording of questions. However, you can re-read them and define or explain a word if you feelthe respondent does not understand it.
Explain that you are going to write down replies,and why. If you read back answers to free responsequestions you assure respondents that they are notbeing misquoted. This sometimes helps to securemore complete replies as respondents may add totheir original remarks if they desire.
Do not be. afraid to wait for. a reply. Time maybe needed to orin thoughts and to think of ways toexpress theni A smile, or "That's all right. Takeyour time in answering," sometimes helps. Be agood listener. Let the respondent talk. Do notrush him but try .todiscourage talk about inciden-tals if they have no relation to the study.
Be careful to follow questions in the order oftheir appearance On the schedille and never omitany that apply to the ..-person being interviewed.However, be alert to notice when a person answersa question during conversation that is to be askedJater.
Replies Must be encouraged without appearingto agree or disagree. Agreement may influence aperson to give future answers in terms of what heor she thinks you want. Disagreement may causea person to 'suppress real opinions. Watch for re-actions from the respondents; make marginal notesabout them.
If interest appears to lag, some statement as tothe time required to complete the interview mayhelp. A change in tone of voice, or a shift in pacemay help. Transition statements froin one areaof a schedule to another also help to .maintain in-terest. For example, if you have been asking ques-tions about clothing, you might say, "You haveanswered the questions on clothing. Now I havea few others about' foods." Or, t`The informationyotf have given us about crops raised -last year willbe very helpful. Our next questions are about live-stock."
In opinion or attitude -questions;it is very impor-tant to assure respondents that ther9 are no rightor wrong answers, Frequently a person will askan interviewer, "Is that what you wanted me tosay?" In . this case, you can say, "Yes, if that isyour opinion. There is noone answer to this ques-tion."
More skill and patience are required to obtainreplies to free response (open-end) questions than'to checkoff questions. A friendly tone of 'voiceand a conversational manner will encourage a re-ply. "I don't know," or "I have no opinion," areeasy ways to avoid expre,ssing real feelings. Aninterviewer must find ways of getting past suchreplies to get usable answers. Sometimes repeat-ing the question in a different tone df voice helps.Sometimes careful probing is justified. :
If probing is used, you must be careful not touse suggestive probes. For example, you mightask the' question, "Why do you like a 15-minuteradio program on one subject?" A suggestiveprobe would be, "More interesting?" This sug-gests a reply, and may be pounced upon by therespondent regardless of what his real reasonmight have been.
Once the person has started -to talk, the inter-
54
fib
viewer may probe for more complete answers bycomments- such as, "Anything else?" "Other ideas?""Another comment?" or "Are you sure that is all?"These do not suggest answers but do encourage therespondent to come up with a more completeanswer.
It should be noted here that there are occasionswhen "no opinion" and "don't know" replies arevalid answers. If a person really has no opinionor does not know, he should not be forced PTV)replying further. More harm than good may bedone by trying to force an answer if the respondentis reluctant to reply to questions. Rather than'antagonize the respondent, the interviewer simplywrites "refusal" or "did not choose to answer" be-side the question.
A final reminder while securing informatkr:Write clearly -and make all entries as complete aspossible. Although they may seem perfectly ob-vious when recorded, sketchy notes, unfinishedphrases, or single words often prove to be meaning-less or ambiguous to the person who must tabulatethe answers. While reading back th'e reply to therespondent as a check on accuracy and complete-ness, try to imagine what a person, who must reactto the recorded entry only,, will get out of it.
Closing the InterviewA pleasant impression of.both study and study personnel should be left withthe respondent. He or she should be left with afeeling of having been helpful and that the coopera-Lion has been 'appreciated. Always extend someexpression of appreciation of the close of-the inter-.view.
Try to have the parting be such that your return,either as an individual Or as a member of this oranother research team, would be welcomed. Youflever know when you may have to call back aboutsome item, or follow through on some afterthoughtrelated to the study. It is a matter of good commonsense and professional ethics to try to preservefriendly relations for future studies, whether byyou or by others; which may involve your re-spondent.
EditingToo much emphasis cannot be given tothe importance of checking the completeness, accur-acy, legibility and consistency of each schedule be-fore you turn it in. Careless editing by the inter-viewer can bias the results of the entire study.Never fail to go over the schedule, either beforeyou leave the respondent or befOre you move onfor the next interview. In this way you can clearup discrepancies and correct them.
.
Be sure all identifying information is entered.Write out abbreviations, complete fragmentarysentences, rewrite' hard to read passages, and ex-pand Eketchy remarks. Make marginal comments
[ 55 ]
to explain apparently inconsistent or unusualentries. Sign the form so that others will know whoto contact for further clarification or follow-uptasks.
Although your work will be edited again, bothby you and by others, the editing yOu. do at thispoint is likely to be the most effective in improvingthe overall quality of the study.
Interviewing Without a Schedule (questionnaireor record form)Most extension studies are madewith the aid of some kind of record form. Wheninterviewers are involved, sonde kind of schedulealmost always is used. However, thire are situa-tions where it is not convenient or advibable to usea standardized record form. Oftentimes, in explor-atory research particularly, a carefully structuredquestionnaire or schedule will hinder more than itwill help. This type of interviewing will not bediscussed to any extent in this chapter.
"Unstructured," "focused," "depth," and "non-directive" are terms applied to interviews madewithout apparent aid of a record form, The inter-viewer, sometimes called the "operltor" in thistype of interviewing as he may work with a taperecorder, dictation machine or similar device, ap-pears to carry o'n aimless conversation with therespondent. He or an assistant records everythingthat is said id analyzes the comments after, theyhave beep typed.
Actually, the interviewer-operator directs theconversations by carefully inserted comments ofhis own so that all the areas he desires to have men-tioned or discussed get worked into the conversa-tions. He may work with a memorized que sheetor he may have a topic outline before him whiletalking to the respondent.
Although more involved, and difficult to trans-cribe, these same techniques May be used for groupinterviews. No doubt, we have all seen themdemonstrated in radio or television interviews whenat least the announcer knows what information hewants to come out during the interview.
Although these types of interviews have theirplace, anu are increasingly important in extensionresearch, they have to be ruled out formuch of ourresearch because more skill and training is required:\of the interviewers. SpeCial probing techniquesmust be mastered to get results without appearingto direct the interview.
Data are more difficult to analyze' in this form,from both.a time and a skill-of-operator standpoint.They do not contribute so readily to statisticalanalyses and, as a result, are limited to preliminaryor exploratory research rather than research toserve as a basis for generalizing.
We are becoming increasingly aware- of a needin extension to go beyond the usual descriptive
65
type of study we-do so often, and work out waysof really getting at what people think, what theywant, why they change, and hew they are motivat-ed. These "nondirective" techniques may be oneswe will have to adapt to our needs if we would keepup with the demands of our work. It is hoped thatmore extension researchers will desire to becomebetter trained in these techniques. At the momentthey are relatively new, but there is some goodliterature avail. ble about them for those who areinterested. If an extension worker is interested inthe type of interviewing and doesn't have the need-ed training, other help should be sought.
Summary
For extension research purposes, we think ofinterviewing as the more personalized ways ofsecuring information, the face -to -face or person-to-person types of data gathering. We also think ofthe kind of interviewing in which a sc+,,-,Ittle orquestionnaire is used rather than when. LEE- more
970
I14;
elaborate and demanding nondirective techniquesare employed.
We believe it is important for a person directingresearch to be aware of what interviewer responsi-bilities are ..and we discussed some of these respon-Sibilities in part. Next we mention some things toconsider in training interviewers and include a sug-gested outline for a six-hour training prOgram forinterviewers.
Then we elaborate on the four steps of the inter-viewing taskmaking the contact, collecting theinformation, closing the interview, and editing theinterview form. We close the chapter with a briefdiscussiop of interviewer"techniques for situationswhere routine interviewing methods are not effec-tive. We suggest some of the reasons why the new-er techniques, for the present at least, are notreadily adaptable to 'extension research and .evalu-ation, but we go on to state why this is the directionin which extension researchers must _explore tofind ways of dealing with the growing complexityof extension problems.
[ ]
c.
v
'- Chapter XIII
DEVICES FOR COLLECTING DATAGladys Gallllp
A good deal of behavior can be observed throughthe senses.; recognize improvement in terms ofaccepted standards in a Club boy or girl -or anadult leader by watching the'person preside or byhearing him talk at a meeting at different times.
We can drive through the country and observethe increase in compliance with soil enrichmentpractices, the use of suggested varieties of grain,the improvement in livestock, and the bettermentin farm and home buildings and yards. We lookfor evidence of favorable change in light of what-we have been teaching. Even changes in intangi-bles such as attitudes,' interests, and personalitytraits are expressed in ovcirt behavior. Such obser-vation is good, but it is nett sufficient.
The Need for Measuring Devices
Different kind; of devices have been designedto.meastire individual behavior. These are deviceswhich for the most part, gather evidence as to aperson's knowledge, attitudes, °I-I:his activities, putin another way, what a person knOws, feels, or does.These devices take but a few minutes to apply,.whereas direct observation of behavior by conipe-tent observers often requires many hours or-daysand is influenced by observer experiences andunclerNndings.
The questionnaire or other device used to eval-uate a program should be 'worked out especiallyfor that particular program. We cannot usuallyexpect to find one ready made because differencesin objectives or in the methods of approach gener-ally make a technique that is effective for one pro-gram unsatisfactory for another. However, beforeconstructing a new device, it is helpful to examinesome of those which have been used previously.They may be a source of suggestions for the par-ticular problem upon which we are working andmay furnish test patterns for construction.
The Selection of the(Measuring Device
When selecting the kind of devices to be used;we should'keep two things in mind: (1) The teachingobjective, and (2) the evidence for-which we needto look in order to learn what progress has beenmade in reaching the objective:
-[ 57
In the educational field there are numerous de-vices that can be used to measure progress towardan educational objective. Some of those which aredesigned to observe and record behavior are:
Knowledge QuestionsL-It is important to knowwhether a person knows information and the levelof information.
Understanding Questions It is important toknow whether a person understands or can applycertain acquired knowledge of principles.
Skill or Performance RatingsThese are used todetermine skill being performed or the productresulting. They'may show the amount of skill thehomemaker has or to what degree skills have beenacquired.
-Attitude ScalesThese are used to show howpeople feel toward things, whether they are for oragainst certain questions, social issues, or problems.
Interest ChecksThese are used to find out inwhat people are interested.
Confidence Testing--A measure of self-assur-nce, the. feeling that you can do a ceaain thing.
Adoption of PracticesThe adoption of a recom-mended practice, is frequently used in extensionresearch as evidence that the farmer or homeinakerha's changed behavior and acquired new knOwledge,skill, and understanding.
Some Illustrations of Devices
Knowledge Questions or Checks on Levels ofInformationObviously it is essential to impartmany facts in order to teach any subject matterin Extension. When we test we can ascertainrather easily.whether the persons taught acquiredthis knowledge. But can be assume that theyunderstand hose facts which they have learned?Certainly we \should not assume-that they are ableto-apply their knowledge. If our objective is todevelop understanding it is necessary to extendour questions 'in order to learn if this knowledge iscompehended.and applied.
In the schedule used in a .Wisconsin Study aresome illustrations of questions which atttnpt toget at knowledge Only.
1. Some fertilizers are higher in plant food thanothers. Which one of these-fertilizers has themost nitrogen? (Check one only)( a. Anhydrous ammonia( ) b. Ammonium nitrate( ) c. 0-20-20( ) d. 10-10-10
2.' The acidity or sourness of a soil is sometimesexpressed as a pH number. Which soil is themost acid? (Check one only)( ) a. pH5( ) b p1I9
3. Acid soils have a number of bad effects. Whichone of these things does an acid soil do?(Check dne only)( ) a. Ties up potash so crop can't get it( ) b. Ties up phosphorous- so crop can't get
it( ) Makes soil hard,( ) d, Keeps soil from holding water
4. How many pounds of nitrogen would there bein 100 pounds of 3-12-12 fertilizer? (Checkone only)( ) a. None( ) b. 3 pounds
c. 12 pounds( ) d. 27 pounds
5. ..Some crops have "hunger signs" which showthat they are in need of plant food. Whenalfalfa leaves have white dots around thmargins and the lower, leaves dry up, whatplant food are they probably short on? (Checkone only)( ) a. Nitrogen( ) b. Phosphorous( ) c. Potash( ) d. Calcium
6. You've probably beard or read that youshouldn't plant corn on a 9f"0 slope. Howsteep is 9c,`,? (Check one only)( ) a. 1 foot drop in 9 feet( ) b. 9 feet drop in 1 rod -
( ) c. 9 feet drop in 100 feet( ) d. None of these
An example of a knowledge question in the field.of marketing information for consumers was takenfrom the Lake. Charles, Louisiana Study.' Thisquestion was designed to get at the level of infor:mation of homemakers in Lake Charles.
7. If you were going to buy a higher .grade cut'of beef, what is the main thing you would lookfor? (Check one only)
y a. Leanness, or free from fatb. Presence of a yellow rim of fat
) c. Deep red in color .
d. Little lineS of fat running through themeat
) e. Don't know
This question was asked-as an open end question.The enumerator Men checked the correspondinganswer on the schedule.
Other examples of knowledge questions are takenfrom the Arkansas Extension Pilot Cotton Project"on disease control.
8. Angular leaf spot is carried primarily fromseason to season by which-of these means?(Check one or more that apply)
) a. On the -seed) J. In the soil) c. Plant refuse) d. Mechanical equipment) e. Don't know
n
Below is a series of questions about common cot-.ton insects. The interviewer.gave the esponclenta picture of a common cotton iiiSectandasked questtions about it The interviewer then checked orwrote in the replies on his record form.
9. What is the name of this insect?( ) a. Bollworm
b. Other( ) c. Don't know
10. At what stage of growth of the cotton plant isit recommended that farmers start sprayingor dusting to control this insect?( ) a. Pre-square
b. Other( ) c. Don't know
11. What is the standard or indicator for deter-mining when -control measu es for this insectshould be applied?( ) a 10 to 25% punctured square dry
year) b. 25 to 40% punctured squarenormal
yearc. Other
) d. Don:I-know12. What kind of insecticide is effective in con-
- (- trolling this. insect?a.
( ) b. Don't know13. How often should this insecticide be applied
to effect continued control of this ,insect?a
( ) b. Don't know
[ 58 ]
6n
14. How much of this insecticide should be ap-plied per acre, midseason; how much lateseason?
Midseasoo Late SeasonDust a. d.Spray b. e.Don't know c. .
IS. When do you quit applying insecticide for con-.
trolling this insect?
b. Don't know
Understanding QuestionsIt is important o findout whether a per' nn understands or can apply cer-tain acquired kr; --xledge.
Comprehension or understanding usually can beinferred from what the respondent says or does.Present Hill) with a new situation and see whetherhe can apply the basic principles taught him. Thesituation must be natural and not so new but thathe can see the similarity to former situations,
There are various levels of understanding anddegrees of understanding. Trying to measure atton high a level: or too ideal a level will be dis-appointing. On the other nand, to nie4ure at toosimple or too low a level not tell you whetherthe person can apply the knowledge at his ownlevel of living and working.
Here are some illustrations of methods'used totest understandirrg:
I. Teaching selection of accessories for the baskdress or suit." The home demonstrationagent or a local leader wears a basic dre!..; ora basic suit and teaches the selection of acces-sories to be worn with it.
After the teaching is finished a secondassortment of accessories, including hats,shoes, scarfs. jewelry,- purse. and the like, arebrought before the group. The members inattendance are asked to select appropriateaccessories for the agent or leader to \earfor two or threr i.1itierent occasions. This willindicate if they know the principles and ifthey can apply them.
2. In teaching farmstead planning" in workshopsto those farmers and homemakers interestedin building, there are certain principles taughtas to location of buildings on the farmstead.
The extensioin vorker asks each husbandand wife present to draw a rough plan of theirfarmstead, showing slope of ground, directionof wind. view. location of highway and drive-way. Then, using circles to indicate build-ings, they put in the house and the farm build-ings, applying the principles of farmsteadplanning. This tests their knowledge of torinciples and the ability of the husband.andto apply principles to their own situation.
3. In teaching art principles" and the applicationof art principles in design and color in thehome at the local leader training meeting. amimeographed sketch of a living reom includ-ing doors and windows is gi% en each leader.Each person at the meeting is asked to a-range the furniture according to principlestaught. Then swat. iie4if materials and wall-paper are used to work out the color schemefor the room.
4. Problem-solving 'devices' can be used to ,meas-ttre understanding.-- The following is an ex-arnpre of the type of questions that can beused by 4-II Club boys and girls:
A farmer wishes to market his spring cropof hogs before the usual drop in market price.Which of the following practices will help?(Check one or more)( ) 1. Have pigs farrowed in March or early
April.( ) 2. Have pigs farrowed late in'May.( 1. 3: Put brood sows and litters on pasture
with self-feeders of shelled corn andtankage in separate compartments:"
( ) 4. Wean the pigs at 6 weeks of age.( ) 5. Provide self-feeders after weaning.
) 6. After weaning, put pigs in dry lot andfeed entirely on corn.Limit the feed during the summer andhave hogs on fall crops of corn.
Revsoas: (Check the statements beloW whichyour reasons for choosing the prac-
i:cc-.. !,ou checked in the preceding list.)a. The usual market drop which affects
the spring crop of hogs starts in Sep-tember.
( ) b. The usual market drop which affectsthe spring crop of hogs starts in No-vember.
) c. Under favorable conditions hogs willbe ready for market at 6 months ofage or less.
.( d. Hogs gain most rapidly when hand fed.( e. Hogs can usually be produced most
cheaply on a limited grain ration forth st few months, followed byhea eding.
( f. Broo sows and their pigs are able to-.13ala .e their own ration-when provid-ed "f ee choice" of tankage and corn.
Skill or perform nce RatingThere are two rea-sons for measuring kills: (1) To find out if the peo-ple have acquired a articular skill that has beentaught and (2) to fin, out how many skills have__been attained.
Skills /1-17,t are important for clientele to learn
59 1
CO
are listed. criteria are set for measuring the skills,and some devices are prepared for use..
Lsuall score cards and rating scales are usedto measure skill. performance. and ability. Theyare used extensively in Extension at fairs and otherevents in judging products and in setting standardsof quality. The product Or skill to be rated is care-fully analyzed. each aspect is considered separately.
It is necessary not to have too many points on ascore card or rating device and there should be adescription of two or more levels of quality for each
of the product. skill, or ability to be rated.;'he examples below arc devices for judging
selected skills in a 441 Club clothing project. Witheach skill is explained exactly what the "rater''should look for in rating that particular skill.
1. .Score Card for Judging Garmenti(Adapted from device constructed by Lois Co.)bett. Michigan Extension Service. for judging441 Club clothing projects.)Club Member AgeLocal Leader Rater
Directions: .1-tate garment on each iten whichappliecto it. rating 1. 3, or 2, depending uponwhether the quality corresponds to the de-scription in the left hand column, or the righthand column, or falls between the two. AddScores at bottom of page.
Two or three items are shown for "constr.:it:-tion of a dress."
STATE-MENT No.. 3 SCOPE
Ski;1 1r;
!ihliwiz
tr. fn.11.
,C.Ir t I. ...t!
Judging Clothing Skills
These arc the instructions to the local ,1-11Club leader: "This chart is prepared for usein judging the sewing skills tht girls aredeveloping in their 4-I1 clothing work.
"The sewing skills of each girl are to bejudged twice, once at the beginning of theproject for as soon as you have an opportunityto observe her skill and make a judgment) anda second time toward the end of the project.
"Make your best judgment of each skill f(.!.each girl using the following yardstick:
Yardstick for Judging Clothing(Five Degrees of Skill)
(1) Practically no skilldoes not l i. Lo',)r.
12) Little skillneeds much hely
[tioj
I3( Fair skillbut still need; somepractice.
i-li Good skillneeds very Lttle,help or prac-tice.
(5) Excellent.skilldbes lot need any-help orpractice.
"In the example: below a7.d on the foil), rigpages the numbers in the squat thenumbers on the yardstick. .',-)12f_ -around the number which she ,s s judg-ment of the girl's skill. The top row of thepair is for' pthe beginning of the projectthe lower row is for thd end. The d;:ferencebetween the two ratings shows Low muen thegirl has developed that skill.
Example: Skill inthreading needles,
2 3
204
4
c
5
(Consider how Beginningthread and needleare held and speed Endin threadingneedle.)
"This. means that. at the beginning of theproject this girl had "practically no ,skill"she didn't know how to thread a needle. Butduring the clothing work she improved untilat the end she had developed "fairbut still needed sonic practice and help.
"Sonic girls in some skills may have "lairskill" (3 on the 'yardstick) to begin with aridgo as high as "excellent skill" (5 on the yard-stick) by the end. Others may make no im-provement at Still others may begin at"practically no skill" and end at "excellentskill." Others may begin the work this yearwith -"excellent skill" and of course end with
x: pent skill.",.A: sure to base your judgment each time
jn your observations of the girl doing theskill and on the quality of work she does.Since some girls may not happen to perform-:.-ome of the skills, you will not be tojudge every girl on every skill. Use a separ-ate pamphlet chart for each gill."
3. _Local Leadership Rating Scales
Extension workers have used different typesof rating scales with local leaders. A paiatrating scale is sometimes used which indicatesa scale of specific needs: 1 to 5 standing forvery much, much, consideraole,, some. littleor none, respectively.
Itor example: How` much help does Cisleader need in conducting a discussion theet-Mg?
:3 4 5
Vertnuett (able Some Little or
none
Amount of Help Lunt Leader :'''(ed:i
Attitude TestingThe extension worker is c-on-corned about the attitudes and beliefs of people.Sometimes people indicate a disposition for or;
,against - ;.:.tinething. or sometimes they indicateuncertainty. A person may be ;:tvorably inclinedtoward high rigid price supports he may favorflexible price supports. en- he may fie in eldubt a-n, which he prefers. Attitudes and beliefs can beidentified only when people are in a situani in-whih they arc confident that they are frve ;t)cxpresss themselves as to what they feel or th lk.
-State Extension Workers' ballots" illustrate ,t.0attitude device employing r»ttltiple choice qt.e-tions. The extension worker can express how hefeels on a number of ilttestions. There may beseveral questions. The_ first question is more gen-era!, the last question more specific:.
The ballot gives a person an opportunity to ex-pess his views on a number of (11..SUOUIS related tothe Extension Se:- vice. Direction.; read as folh ws:
Fl"etocc each question yon wilt fine.: live possi e..._answ7.0.s. Read all five all';Wel*S. and choose theone that comes nearest to expressing the way youfeel about the question. Then vote by putting ,a0.."X" at the 1...tt of the answer you have chosen."
"Vote for only one answer to each question. ifnone of the five answers to a question happ:ns toagree exactly with your own opinion. vole for theone that. comes closest to :ayitin ::ItEtt :,ou thinkabout it If you want to %%The any t:omments onthe balk)'. that will h,. ,.II right; but be sure to vote3n one question by ontkiug an "X." Use the backif the pages for your coil-tun:tits if you need moreroom to v, rite. You do not need to put your name00 the b,llot."
1. !low do you feel aZo-oul the Extension !--;i vice.:::, .tri institution
There are many Se: .all,, vvakne.-,6,:sin the Ev'en-.ien Service.
I I. The Extension Service has ::otne %vein:-ne':.,es, but none are too serious.
I c_ The Extt.nsin.-1 Service: h-'c bout as70any good points as bad omit .
I t d The Extension Servicf has loot :-toodpoints than bad points.
1 i e. The Extension Set ..,. has no seriousweaknesses.
. 2. flow much help di. :.oil get '. too your DistrictSupervisor in doing youe ...ork'.'i ) a. He never helps at all in my work.( i h. Ile hardly ever helps.
a.
) c. Is sometimes fairly helotI (I. Is usually quite ' 1
e: Always gives 17 all '.'tepossibly want.
3. IN 1iv OPINION, separate 4-Ii Chili subject-matter leaflets and bulletins need to be pre-Pared for wm in =1-1.I Clubs, rather than LosingMaterials prepared for adults:
YeS (1):Yes, depending upon age
of members (2.Yes, deimeling upon.
matter t.ontent (31NoNo -opinion
-.I. Now \VG would like to have your opinion onsome ideas. I shall read eight statements toyou and would or you to tell me whetheryou agree or disagree with the statements..You may not have an opinion cort some of them.
AGREE DISAGRFENO Or INION I
ORUNDECIDED!
ST AT IAENTS
I, i :is 1, 11111.0 1 .11t, 1,1 1.1111, ,1.
11,.i, a I .i.11; !
I. Itt. ....I 4,, 1,4. .1, '011, ".
711.1N . IP;ri
-tr.;1!.t 1....
T. ti is
1.11d
41',1111
11 1.111
1.1
..,r I.
1
.111
1.11111,t 11111.1.11, !It .11111 ;:1,11
Intirest 'Checks An extension program maysucceed or fail in proportion to the emphasis whichthe extension vvo-ker puts upon the developmentand encouragement of interests. Interests indicatelikes and disl:i:e3, stfiztf:aii.tott, and dissatisfactions.
Interest cheeks, Devices for lindinZ out in whatthe people arc interested, have been used to con-siderable extent in F,xtension.
The ''`Suggested Form frr am Interest Question-naire" illustrates a ,itethoct of finding out what helpre :.41 people art-, interc,ated in getting from the(xi" n ly agent. his is used as a partial basis forprogram planning.
131
1.4 It
r
1 Form for an Interest Cli.:1,..V:71:-naire`
In the following list of Ulm's. the interviewerwill check fol each Dv::: degree to which thepersonnterviewed is inici-ested in receivinginformation or other assistance, and methodspreferred.
"Much interest means: Ask yourself wheth-er you would read about it right away if youhad the reading material. or if you wouldattend a meeting with other farmers to hearthe subject discussed. If you would, thencheck in the "Much" column, meaning muchinterest.
"Some interest means: If you are not sure,or if you wouIdpr ably put off for some timereading about it r attending a meeting tohear it discussed, en check in the "Some"column, leaning so erest, but not much.
) c. Not very well satisfied) d. Not at all satisfied
( ) e.Any corn,ents
Ti above question is ne that has been used tosatisfactions.
Confidence TestingSelf-assurance or self-confi-dence, feeling that one can do a certain thinghas in the past been considered too intangible tomeasure. And yet you ! probably observed aperson and noted that "he is very confident" or"he lacks confidence." In so doing you have ob-served and d: awn a conclusion; you have "measur-ed" his confidence. Therefore, there must be tan-gible evidence of confidence.
The following is a device or question which hasbeen proven vplid and has worked successfully. Itcan be adapted to measure confidence in manydifferent kinds of things.
Have you ever made a school dress for yourself?7' ( )4a._ Yes
( ) b. No
Do you think you can? (Check one)) c. No, I don't think I can.) d. Yes, I think I can, but am not sure.) e. Yes, I feel sure I can.
Practices Adopted or Action Taken Perhapsmost of the devices and questions used in Exten-sion to measure change are aimed at finding out ifthe person has adopted the practices taught. Apractice adopted is also evidence of knowledgeacquired and attitude changed. In time and energymanagement certain principles are taught such as"sitting to-work" and "eliminating some jobs."
Following are examples of two questions taken"If you feel sure you would not read about from a check -sheet prepared by a home manage-
the subject or attend a meeting on it, no mat- ment specialist. This check sheet was designed toter how much t.. ne or reading material you be used before and after the learning experience..have, or how ma :y meetings were held, then ..,
.4,....---A4rWre are several different things whichcheck in in the "None" column, meaning nointerest. 0"- pie do around the house. We are interested in
"Be careful to consider and check every finding out how many vi,,omen are doing thesethings. So I would like you to go through thestatement, but do not spend too much time an list and check which ones you do."any one statement."
2. Hol ng have you lived in this community?a. No. years_
3. liow satisfied a e you with the way local gov-ernrnent affair are handled? Wouid yc..t sayyQ11 are not at 111 satisfied, not very well satis-fied, pretty 'w 11 satisfied, or very well satis-fied?( I a. Very well satisfied
- ( ) b. Pretty well satisfied
Home ec,momics example:
1. In ironeLne.,4o you:
a. In towels?'14.441'---4Iron sheets?
2. Do you sit to:a. Iron? _ _b. Prepare vegetables? ('.),___(2)____(3)._ _c. Wash dishes?
Usually Sometimes Never
(1) (2)_ __(3)
Agricultural example:3. Do you sell-eggs on a graded basis?
( ) a. Always( ) b. Sometimes( ) c. Never
4. Where are eggs kept until marketing( ) 'a. In the house( -) b.-Where-( ) c. Outbuilding (specify)
Practices Adopted and Methods Responsible-Many of the ,questions determine practices adoptedand attempt to determine methods the respondentthinks are responsible.
1. Do you have electricity?Yes (1)____ No (2)
2. Ifnes, a. Do you, yourself, repair yoii.ir ownelectric cords or plugs?
Yes (1)___ Nob. If you repair cwn cords or plugs, where did
you learn to do it"'
Summary
Devices for collecting dr.ta are forms on whichthe dat' can be recorded and used at a later timefor ant.Q,sis and tabulating. Memory does not needto be depended upon.
In selecting a -device we need to keep in mindthe objectives to be evaluated and the evidence tolook for. We have given some examples of ques-tions and forms for eight kinds of objectives. Manytimes. however, we must construct our .own deN'ices.This is the subject of the next chapter.
IiReffiencesr.` Cooley, E.; Powe, D. D.; Matthews. J. L.; and Ueland, G.
Informing Consumers in Lake Charles: An Evaluation.Baton Rouge, La. Agr. Ext. Pub. 1233. 1958.
2. Factors Infly tc-iig.,\Knowledge of .Technical Soils Con-cepts by Wisconsin Farmers, Bulletin 27, College ofAgrstIV, Univeisity of Wisconsin, March, 1957.
3. Fes n, J. G. "Tools for Testing." Extension Serv,ice Review, September, 1956, pp. 178-179.
4. Fleck, H. .% How to Elialuate Students. Bloomingt..7.,Ill. McKnight and McKnight PublishingCo., 1953.
5. -ruichey, F. P., Deyoe, G. P., and Lathrop, F. Ws.-"TheMeasurement of Understandingein Agriculture." The
.... Measurement of .U.n de rstandiria. Chapter xiy,__15thYearbook, Part I, National Society for Study ofEducation,'1946:
6. Frutchey, F. P. and.Church, H. L. Educational Growthin the 4-H Clothing Project, Missouri, USDA, Ext. Ser. .Cir. 382, Washington, D. C., 1942.
7. Frutchey, F. P. and Forbes, M. E. Evaluation in the4+1 Clothing Project, Massachusetts, 1940-41, USDA,Ext. Ser. Cir. 384, Washington, D. C., 1942.
8. Leagans, J. P. The Educational Interests of Farm Op-erators in North Carolina asultelated to Work of theAgricultural Extension Service. Ithaca, Cornell Uni-versity, 1948.
9. Noyes, IL; Dickerson, L. B.; and Lusk, M. Examplesof Problems or Situation Statements Used in Evalua-tion. Extension Service, Washington State College,Pullman, Washington, 1955.
10. Parten, M. B. Stirve's, Polls, and Samples. New (ork,Harper and Bros., 1950.
11. Raudabaugh, J. N.; Saunders, J.; Cavender, J. Arkan-sas Extension Pilot Cotton Project. (In progress.),
12. Rothney, J. W. M. What Research Says to the Teacher.No. 7. "Evaluating and Pupil Progress."National Education Association, Washington, D. C.,1955.
13. Sabrosky, L. K. Six Keys to Evaluating ExtensionWork, USDA, Ext. Ser. Cir. PA-371, Washington, D. C.,1y39.
14. Sabrosky, L. K.; Lo::::Aris, J. P.; Gallup, G. ExtensionEvaluation, What Is it. How It Is Done, Its Use. Out-line for Summer School Course, including discussionsand studies di-',eloped. Colorado A. and M. College,June 24 -July 13, 1946, USDA, Federal ExtensionService.
Nfat,ria I in ilk chatr is 'bawd oil at t "Stlet.1 iorr of the Aleasiir-itilz k. Sabre)sky in It.tereitet. No. /4 aboveAlso OH. di:tut ,r it basd sok.,111,,ot disen.s.Nions Kith :Mr& tinbroskyand othcr tionlhers of t ht. Iritiaon of 11toNitoil livto..art.11 and Train ins:.
[ 63 )--r)4-0 .
Chapter XIV
CONSTRUCTION OF ANEVALUATION DEVICE
Gladys Gallup
The selection and construction of the measuringdevice depeno.3 upon the kind of evidence wantedin light of the objectives of the study. At thispoint the evaluator needs to think ahead how heplans to collect tits: information.
The county vricui agent may decide to usean observation'sheet t .-ry with him as he makesvisits to a cross section of farms and homes. fiemay decide to send mail questionnaires to a sampleof farmers or homemakers. The home demonstra-tion agent may develop report forms such as check-lists or questionnaires to use in collecting informa-tiora. .regarding the adoption of practices from asample of participating horOrnakers.
Sometimes a field study is made. This is a moreformal *evaltfation study. It" requires a ("fferenttype of question afire or schedule as it is to be usedto. interview a cross section sample of the peopleliving in the county. You need, therefore, to con-sider not only the kind of evidence which youwant to meet the objectives of the study, but alsohow you plan to collect the information.
The questions asked should be so set up thatinformation or data collected separately from anumber of people can be added together or sum-marized to present a true picture of the group.
The questions in the measuring device should bedefinite and specific questions related to the majorquestions which are to be answered by the evalua-tion. For example,' if one of the major questions tobe answered by the evaluation is "What hag beenaccomplished to control soil erosion on the farms,"specific questions relating to soil erosion controlwould be constructed for the questionnaire.
When the replies to these specific questions aresummarized, they answer the major questions onsoil erosion control. The major questiOns are the -main pUrposes of the evaluation and the specificquestions of the questionnaire are designed in rela-tion to the main purposes of the evaluation.
Criteria for Constructing a Measuring Device
For a device, such as a report form, an interestcheck, or a questionnaire to be effective there arefive criteria which it must meet. These criteria are
Validity is concerned with measuring what youwant to measure. A valid device actually gets evi-dence of progress in the teaching objectives or ofthe information yowdesire. The two elements ofvalidity are behavior validity and subject-mattervalidity.
Behavior V:lidity-7A device has behavior validityif the behavior called for in the questions is thesame as that expressed in the teaching objectives.The device gets evidence r f the behavior you areteaching. Measuring skill, for example, will re-quire a different device from measuring knowledge,because of the different behavior involved.
Subject-Meter ValidityA valid device mustdeal with 11.X vallicular subject matter of the teach-ing objectis.re it is obvious that if we are measuringattitudes towa-d. soil erosion control, we would notask questions about attitudes toward using ferti-lizer. The device must contain the subject matterand situations in which the behavior is expressedas called for in the teething objectives.
When a device is valid (corn the standpoints ofboth behavior and subject matter, valid evidencecan be obtained from which to draw conclusionsand make recommendations about the teachingobjectives, methods or other aspects of the programwhich we are trying to evaluate.
Reliability is the degree to which the sample of'people or of subject matter represents and givesresults which are like the results that would havebeen obtained from the whole population of peopleor of all the subject matter. It is a measure ofwhether the random sample is large enough togive sufficiently stable results. The sample maybe a sample of people, a sample of subject matter,or a sample of behavior -tver a period of time, de-pending upon the "population" about which inter-pretations, are to be made. Extetnsion research hasbeen concerned mainly with samples of people.
Objectivity means having a concrete, observable,and tangible basis for making a judgment or inter-
[ 64 1
pretation. It is in contrast to subjectivity, 1)inior-;,or guesses. An objective measuring dc ,-icc is soconstructed that the results from it not influ-enced by the difference in people who administeror interpret it. If equally competent people ask aquestion, interpret -the answers, and get closelysimilar results the measuring device is consideredobjective.. The use of numbers helps to:make Iter-pretations more objective. For example, "100 per-sons" is more objective than "many persons."
-Pia6ficabilif-V=A-ii ea-Suringdevice-imustbe-prac,----tical to use. It is necessary to consider time, cost;and convenience. Will the form take more timeThan it is worth; will it cost inore than you canafford? It must be within your resources consider-ing all the demands made upon you. In planningthe time schedule, the length of the questionnaire,the time involved in suhamarizing it, the coopera-tion you will need of the people and the probable-value of tiTh results obtained continually must bebalanced' against its costs.
SimplicityIs the report form or -ituestionnairesimple, easy to understand, easy to answer, easyto administer, easy to tabulate, and easy to sum-marize? The more simple a questionnaire;the morepractical it will be to use and the time between fieldwork and the report will be minimized. Simplicityshould not be confused with brevity.
'.Che degree to whit a measuring device meetsthe criteria is conditioned by thecircumstances of;.11P study and the use to be made of the findings.
Types of Questions
In wording questions for the different measuringdevices used in Extension simple types of questionshave been used:
. .
Dichotomous Questions. Dichotomous meansdivision into two parts or division by pairs.
These questions present opposite choices design-ed to get response such as yes or no, agree or dis-agree, approve or disapprove.
The chief advantage of dichotomous questions issimplicity. They take relatively little time to ask,and give a clear-cut answer which can be retordedand tabulated easily.
The follow_ing are typical dichotomous questionsfrom extensioNjudies:
Do you happen to know who is the county agri-cultural agent? Yes_ __No__ _
Although these questions are designed as "dicho-tomus" they usually should and do allow for athird response, such as "no opinion," "not sure,"or "don't know." For example:
During the past year have you attended:
A women's extension club meeting conductedby the county home agent?
rememberThe chief advantage of the dichotomous question
is that it can be answered with a concrete, definiteanswer. It gives a clear-cut reply which can beeasily recorded and tabulated.
One of the limitations of the "yes" or "no" ques-tion is that a slight misunderstanding of its mean-ing may result in a complete removal from a positiveto-- a-negative' -answer: This-may-also increase -the-"no answer" rate. This is especially true whengetting a person's opinion.
Multiple Choke Questions are used extensivelyin extension _questionnaires. A direct question isasked followed by.a number of possible responses.The person is asked to select from several Clearly
.stated alternatives.Multiple choice questions are used when there
can be several responses to a question. The personaswering these questions is given an opportunity
/Of choosing one or.more of several possible answersthat are true, or represent his opinions or ideas, orcome closest to them.
The directions may indicate that only the correctresponse should be checked or the directions mayindicate that the respondent can check as manyresponses as he believes apply.
Multiple choice questions are not easy to con-struct. The possible answers must be plausible andthe right answer must not, be overly conspicuous.Example of multiple choice questions where oneanswer is correct:
Which one of these plant foods put on as com-mercial fertilizer will "leach" or wash out of thesoil most easily? (Check one only)
nitrogenphosphorouspotashcalcium
An example of multiple choice checklist type inwhich one or more answers nay, be checked:
Which of the following methods do you use forkeeping up with your farm business? (Check asmany as you use)a Keep bills in a drawer, envelope, or some
convenient place.b. _ _Keep check stubs.c. _ _Keep a ledger (book) which f developed
on my own.d. Keep a farm cash account book (printed
form).e: Keep in mind my expenses and receipts
and add them up at the end of year.f. Other:
Open End Questions,-These questions are usual-ly- used in persbnal interview schedules. The re-spondent is free to answer as he likes. Open endquestions give such a variety of responses that thejob of classifying and analyzing is both time con-suming and difficult.
Open end questions are often used in pretestingto get a line on- possible answers for an eventualcheck list type of question.
When it is necessary to employ questions whichgive the respondent free latitude in his responses_ ,
opeh end questions are used. Example:1. How does your milk production in October,
November, and December compare with othermonths of the yearis it higher; lower, orabout the same? (Circle one)
IF "HIGHER" OR "LOWER"2. Abbut how much (higher) (lower) ?
(percent)
IF FALL PRODUCTION IS SAME OR HIGHER3. Many farmers are not able to keep their pro=
duction up during these months. How do youmanage to do it?
4. '`here did you get the idea"
IF FALL PRODUCTION IS LOW COMPAREDWITH REST OF YEAR5. Have you ever heard of the idea of making
your milk production even throughout theyear by havinr-cows freshen at differenttimes? Yes No
6. If yes, what do you think of the idea?
Wording of Questions
The wording of questions affects* the answers.Each question must be worded to give exactly theinformation desired. Go over each question andask yourself these questions.
1. Is the question related directly to the purposeof the study?.
2. Can there be any doubt about the definitionof any word in the question? Will each re-spondent understand each word as you do?
3. Are the words in each question familiar tothe person who is to answer?
4. Are_ questions simple, short, clear, and con-cise?
5.. Have you avoided asking leading questionsquestions worded in such a way as to suggestanswers?
*'?
6. Have you avoided "double - barreled" ques-tions, questions that have more than one idea?
7. Have you avoided double negatives?8. Are definite time limits nn gut- itions specified,
for example, "During t past 12 months?"9. Has a check yes and no answer been used
whenever advisable?10. Is there an opportunity for the respondent or
cooperator to indicate that the question doesnot apply to him or her?
Face or Descriptive Data Questions
Besides the questions 'relating to the objectivesof the work being studied, additional-informationmay be needed about the respondent which willhelp in interpreting the answerssuch items asr:;unber in family or place of residence.
Certain "face data" questions which can bechecked against census data have been used inextension studies. These include: Formal school-ing, place of residence, age groups, total net annualfamily income, and certain items on the Sewelllevel of livii:g scale such as running water in thehome, complete bathroom, and other conveniences.
These face data questions provide informationby which the study data can be classified. Forexample, we may want to know if the people whoadopted certain practices live on a farm or not,if they are older or younger: or 40 years of age andover, or under 40 years of age, if their formalschooling was 8 years or less, or more than 8 years.(A much more detailed classification mat edesired.) By classifying the information in ti:way we can find the relationship between differentfactors and changed practices. Examples of facedata questions:
Please check the HIGHEST grade you completedin school:
Elementary schoola.' 1st to 4th gradeb. 5th to 6th gradec. 7th graded. 8th grade
High schoole. 1 to 3 yearsf. 4 years
Collegeg. 1 to 3 yearsh. 4 yearsi. more
The questions of this type that are to be includedin the questionnaire depend upon the pur:)ose ofthe. study and how the data are .to be analyzed. \Other examples of face data are:
[66]
7G
Where is your home located? Find the one ofthe seven following answers which best describeswhere you live. (Check one only)a. On a farm or ranchb. In the country but not on a farm or ranchc. On a-farm or ranch part of the year and
in town-part of the yeard. in a town or village with less than
2,500 peoplee. In a town or city with 2.500 to
10.000 peoplef. In a town or city with more than
10.000 peopleg. In a suburban area (not in city limits)
of a city with 10,000 or more peopleIn which age group do you belong? .Please find
the age- group below in which you belong. (Checkone only)
a. 19 ye; 's or under(\h. 20---2, years
c% 25-29 yearsd. 30-39 yearse. 40-49 yearsI. 50-59 yearsg. 60 years or olderPlease indicate which of the following is. nearestyour net income after farm or business expensesare deducted. Check one only)a. Less than $1,000b. $1.000 $1.499-
$1.500 7 $2,499$9..500 $3,499
e. $3,500 $4,999f. $5,000 $6,999g. $7.000 $9.999__
. $10.000 or over
'Core Questions
There is a series of questions that has been devel-oped and used in extension studies to determinethe knowledge which people have of extension,level of living items. and face data questions.
There are many advantages in using these corequestions as they have been pretested under differ-ent conditions and are well designed. Most impor-tant, they provide comparable data because thequestions are identical in wording. This insuresthat the results of a study will be more meaningfulbecause they can be compared with the findings ofother related studies. .-
In addition to the face data questions, a series ofquestions has been developed to determine knowl-edge of and participation in extension. Examplesof such questions which deal with 441 Club workare:
g you know something of 4-11 Club work?Yes. _
no you have children in 4-II Club work?Yes. . No
f. 677
. Have you ever had children in 4-H Cluh wok?
Have you yourself ever been a member or leaderof 4-11 Club?
Another example of a core question as used inhome economics nutrition studies is:
Do you yourself drink milk? About how mucheach clay?a. One glassb. Two or more glassesc. None
Sequence of Questions
Logical Order--Place the questions in the logicalorder so that ;.1 person reading them does not haveto- jQmp from subject to subject in order to answerthem. When the data are to be obtained by theintervieuethod, the questions should be groupedso that the conversation will lead logically fromone question to the next.
If general and :specific questions are used, thegeneral questions should precede the specific ones.;otherwise, the respondent is likely to answer thegeneral questions in terms of the earlier specificones. For example, a general question as "Duringthe past year what. changes have you made in yourhouse, such as remodeling ?" should precede themore specific question, "During-the past year whatchanges have you made in the kitchen?" You donot want the respondentto think about kitchensonly and not mention other more general improve-ments.'
Psychological Qrder You can harre a logicalorder of questions but the opening questiontstouldalso In.., c human interest appeal. Theope'ningquestions should lk easy to answer and of interestto the respondent.
A few simple questions at the beginning of thequestionnaire will give the respondent confidencein himself. This is invaluable for carrying theinterview through later and more complicatedquestioning. Questions of personal nature, likeasking for the age or income level of the informantshould be placed near the end of the questionnaire.
In designing mail questionnaires the arrange-ment of questions in order of interest is particularlyimportant.
Mechanical Setup
Four Parts to 'a QuestionnaireEach well -plan-ned device or questionnaire has four parts:
1. Identifying information such as name of ques-tionnaie or study, county, date, and name ofperson filling out form. If the study is of the
more formal type in which schedules are usedby personal interviewers, the schedule shouldalso include number of-survey area, name ofinterviewer. and place for checker's initials.Also the schedule should include final dispo-sition such as: O.K., refused, incomplete butclosed, etc.
2. The introduction is important because it tendsto orient the respondent to the p.,rpose of thesurvey and tries to Motivate answers to thequestiOns. When records are gotten by mail,motivation is especially important. The open-ing; sentences should explain:a. By whom the study is being Made.b. What is the purpose of the study?c. To,what use the information will be put.
Adequate response depends largely on how well-stated are the facts of the introduction. Anyspondent is due the courtesy of an explanationcovering the above points.
3. The content questions based on objectives.4. Face data, if face data are necessary.
Physical ArrangementThe following six sug-, gestions are on physical arrangement:
Number each item consecutively..Never repeat a number on the schedule. Thisfacilitates reference to items where instruc-tions and definitions are given.
3. Have each item clearly diStinguished for tabu-lation purposes.
4. Vary the type within a question to emphasizethe most important words or phrases.
5. Make form easy to read and fill out. Haveplaces fqr checks rather than for words even
9
as short as yes and no.6. Place spaces for such checks in vertical col-
umns along the edge of the sheet to makesummarization easier.
Adequate SpaceWhen you expect informationto be written in, provide adequate space for-it. Donot expect peopleto use margins.
Good AppearanceThe form should have goodappearance. This can be accomplished by goodpaper, large print, wide spaces, .and good arrange-ment.
Special Rules for Constructing theMail Questionnaire
1. Appearance. The questionnaire should beattractive. This can be accomplished by ap-propriate use of colors, pictures, sp,i,:ing. type,arrangement. and so forth.
2. Length. The questionnaire should appear easyto fill out and be as short as possible.
3. The letter of transmittal or cover letter sentwith the questionnairoshould have a personaltouch and an appeal which seems likely tobring the highest proportion of returns asindicated in the pretest.
4. The letter should be frankable. With eachcover letter and questionnaire there should bean enclosed franked envelope addressed to theextension worker for the respondent to returnthe questionnaire.
Pretesting the Questionnaire
Every device used in measuring should be pre-tested. The way to.1.?arit whether the test is readyfor use is to try it out. The pretest- helpsyou toknow if the content of the questions and the formof the questions are satisfactory. There: are twoimportant points:
1. Try the questionnaire with persons who re-semble those with whom you are finally goingto use it.
2. Test the questionnaire under conditions simi-lar to those in the study. For example, whenpretesting a mail questionnaire have personsanswer if without help from you. 'Tell themyou have a rmestionnaire you wish them totry out an they have finished ask foropinions ano Elestions.
From a careful pretesting you get informationabout the following: How long it takes to fill outthe questionnaire;. whether the questions followeach other in natural order;-whether ample spaceis provided for answers; and whether the directions,are understood.
You have to deterinine how widely you pretestthe questionnaire, and how often you revise It, interms of importance 'Of the study, the precision youneed, and the amount of time and effort you canafford to spend upon the questionnaire.
Summary
The questions in the evaluation device should bedefinite, specific and related to the major questionsto be answered by the evaluation. For a devicsuch as a .questionnaire to be effective there arefive criteria which it must meet. These criteriaare validity., reliability, objectivity. practicability,and simplicity.
The wording of questions affects the answers.Each question must be worded to give. exactly the .information desired. SpeCial rules-fee-Constructingquestionnaires should be followed.
[681
1 0
`References
1. Good, C. V,. And Scales, I). E. Methods of Research. NewYork. D. AppletonCentury Co., 1954.
2. Hawke:, FL. Lindquist,..E. .1., and :1dann, C. It The Con-struction of Achievement Examinations. New York.lIoughton. Mifflin Co 1936, Chapters 2 and 3.
3. Jahoda. M.: Deutsch. M.: and Cook. S. W. ResearchMethods in Social Relations, Part Two: Selected Tech-niques. New York, The Dryden Press. 1953. Chapter 12.
4. Parten. M. B. Surveys, Pods and Samples. New York.II:trper and Bros., 1950.
5
[ 69 ]
i J
5. Payne. S. L. The Art of Asking Questions. Princeton.N. J., Princeton University Press, 1951.
(3.'Sabrosky. L. K. Six Keys to Evaluating Extension Work,USDA. Ext. Ser. Cir. PA-371, November 1958. U. S.Government Printing Office, Washington 25, D. C.
:LI'. 1,11 t I h.I. ,I hap! r, : j thNI, 11. t 111 It 1, in "I'm tr,.11
,tt, Ill .j \Lit. tr 's;tuttrit -,1.1 t. I i t..1 .r lit tiietl,,ion;
:17 1. .1' .1 I.. rt . I I I I , .t I .t. ;ti Art It an dI t
-KV
Chapter XV
TABULATIONWard F. Porter
Mrs -le well G. Fessc)alei
Tabulation is the process whereby study data are tabulations and can work ,ont extensive cross-organized or grouped according to plan so as to tabulations. In such cases, it is essential to includefacilitateanalysis and interpretation. Most of us the machine technicians in our planning. Sinceuse-the-simpler -ta bttla ti ng,--m-tal lying-teehn any-schedules-a re_ttpm-eocled,!1_the_assista n ce-ofquite often in our clay-to-day extension tasks. We machine operators may be required well in advancenote the number of incoming and outgoing tele- of actual data collection.phone calls that are inade at the office, We keel) If we will have to do hand sorting and countinga running total on hoW many publications we distri- 'with or without assistance; we cannot hope to takebute and what kinds. We have a tally sheet for on so large a program of analysiS as we would if weoffice calls, farm visits, meetings with' extension could count on _electrovic equipment. The problemgroups, liaison actions with other organizations, and that could be worked tip in a few days with a trainedthe like. We have a frequency distribution of the, .crew of machine operptOrs might take montlisortopics we deal with on our radio or television pro- be impossible to accoffiplish, using.hand methods,grams Or in our newspaper or magazine columns. before the findings would be obsolete.
When it comes to evaluation work and. research In addition to physical facilities, we have to con-in general, it is obvious we could not get the job Sider certain human factors, smell as our own.abili-done without resorting to tabulation(' We might ties and those of persons who will be at our disposa)not have, to work out' elaborate tally sheets or in- for the study. If we have no competence in statis-volved coding plans, but we must.clo some counting tics and have difficulty iri following through onor we can make no pretense of having clone an involved relationships, we may have to settle forobjective study. reporting totals, working out a few percentages,
and possibly calculating some averages. If we havePlanning'for Tabulation some competence in such matters, or someone is
available to guide us, we can work out a more corn-We should work out our ation plans prior to- prehensive plan of analysis-ranking, correlating,
the completion of the questionnaires or other record intercorrelating, testing for significance, validity,forms. We have already pointed out that we need reliability, and so on.to know what information we want before we canbuilt! a schedule or questionnaire to secure it. 2. What Information Will We Need for Our
of the schedule or ques-mence after we decide what information we wantThinking about the-tabulation should properly com- Analyses? The purpose'
tionnaire,' of course, is to secure the informationand while we are working out practical and objec- we need. However, we can be more certain it willtive ways to collect the necessary data. secure the information we need by drawing up some
In planning for tabulation, we should keep four tentative tally sheets and work tables, preferablymajor considerations in mind. before the fiflal draft of the' schedule. This will
_000' help us see, for example, whether or not the prob-1. What Resources and Skills Will Be Available able answers will be in a forM or of a type that will
to Us?; We cannot plan for a very comprehensive facilitate the analyses we wish to conduct: Oncetabular analysis if we must have a quick ,answer or we are satisjied in this regard, we can make up theif there will be limited clerical and secretarial help. final tabulation forms, and be ready to make entriesIf we are involved in the kind of study wherein we as soon as the data are received.have to make our own analyses at odd moments We must. remember that we cannot plan for anstolen fioni our regular tasks, we have to be satis- analysis that require,- more detailed informationfled with relatively brief and simple study designs, than can be supplied by our record forms. Forrecord.forms, and easily tabulated answers: instance, we cannot make year-by-year breakdowns'
Where machine facilitieS and operators are avail- in our tables if our age data, are collected in termsable, we can plan much more involved and elaborate of five-year intervals. Nor can we readlly report
[ 70 ]
average or ,mean income if we collect income datain terms of categories such as under $5,000. $5.000through $9,999, and $10,000 or over.
On the other hand, however, we can make ourtabulated information less detailed than that report-ed in our source. We may collect age iniHrmationby age at last birthday and report 'it ',ur tally.sheets by five-year intervals. We may have a com-plete age-sex breakdown but report in terms ofnumbers of achilts or children per household. Wemay secure actual income data (very difficult to do)but report in frequency distribution by S5.000intervals,
3, What Relationships Do. We Wish to Study? Wemust .decide on what relationships we plan to studyart h at-we Crave the plc) pe r questions-in-our-data-gathering instill!in our tabulatingin categories suit
TO illustrate: I
ent and have adequate provisionsforms to record the relationshipsble to -our needs.we want to know the differences
in interests of respondents by age, educational level.place of residence, and sex, we must be able to sorton these items and then tabulate in terms of theiranswers regarding interests. In our tabulationpianS we must decide what age, residence, andeducation,categories will serve our purposes, andthen be sure our source reports the information inenough detail to set up these categories.
4. What Comparisons Do We Expect to Make?Whenever we intendto compare certain data fromour study with 'that of other studies, we must becertain to collect the -necessary information andassemble it in such a way that there is compar-ability. Quite frequently we like to check on theprobable representativeness of our sample by mak-ing comparisork)vith census or other statisticalinformation related to our study area. Wheneverthis is our intention, we must be sure to have ourbackground information (our "face data") definedand classified so as to match that used by the cen-sus, or other sources to be usedin'our comparisons.
is a strong argument for collecting our basicinformation in Auite detailed fashion. Then we areat liberty to make a wider variety of tabulationdecisions. We can group our data in the conven-tional categories used by the censtis and similargovernmental agencies to make comparisons withtheir findings. Then, if needed, we can makespecial groupings to permit comparisons with find-ings of special studies by colleagues in Extension
"or elsewhere who have done work similar to ourown.
Another common occasion for comparisons is inbeforeafter or trend studies.' When you arestudying the change that has occurred` since someprevious reporting date, or when you are observingth.'e effects of an action program, you have urgent
need for comparable data. Whenever this is you!'objective, you mustask questions and prepare tab-ulations in essentially the same, form for bothoccasions.
Yon cannot ask for annual family income onetime and annual farm income the next. Nor canyou ask for income to the nearest $1,000 one timeand then decide to simplify the item next time bysetting up $5,000 income categories.
Tabulation Techniques
Tabulation consists of listing or grouping answersfrom all respondents, or from all members of par-
s ticular groups within the population being studied,so that_suminary_ans.svcrs_ean_beobtained.: Possibletechniques for assembling these summary, average,or consensus answers vary widely. The ones youselect will depend on the -kind of study you aredoing, the resources that are available, and whatyou want to get-out of your data. We will mentionsonic of the techniques that you might use inextension evaluation.
Tally SheetThis is simply. a counting aid, al-though it can be elaborated into a sorting device.On a card or sheet of paper of sufficient size, youset up column headings according to the 'categoriesinto which yon-wish to sort or subdivide your data.Down the left side of the page you list the possibleanswers to the question.
CountyPerson(s)Yallying
ba to
HAND TAW( SHEET (EZample)
Development Survey
7. Level of livifig:
Total households
NO- r,. ,onse
Reporting
a. Electricity
1). Telephone
c. Running water
d. Central heating
c. Electric washer
f. None
Allhousehold
Totalfarm
Totalnonfarm
44 25 19
3 1 2
41 24 17
- 40 (23) (17)tY4 f14: 1141 RV MI Mg MAI //
26 (12) (14)fhl 1104/ Il nil au
25 (10) (15)ill4 Ma Oil PH/
19 ( 6) (13)r,f4 Mil Mt ill
27: (14)FM WI It 1 M1 MI Am
0- ( 0) ( 0) #
In.lour example, the sorting factor is residence(farm or nonfarm), While the question concerns thelevel of living of the families in the sample:
We will assume the respondents (or the inter-viewers) were asked to check only. the 'facilitieswhich were available in the homes.
Since it's necessary to account, for each and everyrespondent, alloWances have been ruade for non-responses; and for households reporting none ofthe items listed. This "rule" applies in general toall tabulations.' For certain types of questions, theremight be justification for -still another categorythat is, one to account for respOndents,who reportnot knowing or "no opinion" or "don't know:: Thc:importance of this consideration has. already beenstressed in our discussion of questionnaire con-struction.
To construct such a tally sheet, you reserve spaceat the top for identifying information. Dependingon the circumstances, such identifying data mightinclude: (I) working -title of the study. (2) working:stitle of the particular table (tally sheet), (3) thename or initials of each person working on thesheet, including counter, reader. checker, and anyothers, and (4) date work was completed. lt,is agood idea to type the column and row headingsand draw in ,necessary guide lines.
You can make-the tally with a typewriter, if youhave a wide-carriage machine;_by having one per-
k son read off the numbers .and where they shouldIgo, and" the other enter them with the typewriter.The more conventional way is.tO draw up the tallyform and then make the entries by hand with penor pencil. You can still mlike typewritten copy foryour permanent record, however:
In tallying, it is customary to make four perpen-dicular marks and then make the fifth one a slashthrough the previous four." This facilitates countingby fives and makes it easier to observe relationshipsby inspection rather than by actually counting.Subtotals can be placed inside brackets at the ,endof the Count or inimediately above or below.
This technique is most effective when only.a fewrecords are to he-tallied. and when there is not a'great amount of subsorting. The greatest disadvan-tages of this technique are the likelihood of errorand the difficulty in checking for error. About theonly way to checlthe work is, todo it over, or havesomeone else do it over-and compare results. Event'the same results do,not rule out the possibility thatcounteracting errors have been made. The pros-pects for error increase Materially. if there areinterruptions during the tallying process.
Although tally sheets are most useful in limited,.small-scale studies, there are at least-tWo occasionswhen tally sheets have a place in the more complexevaluations or 'surveys. First, they can be used toadvantage in taking off key items of interest in a
,preliminary, tentative, unchecked sort of way. Suchdata should never be reported or published, butthey can be used to get an idea of what the resultswill be.
.second occasion,whereonly
can be useful iswhen the. researchers have only vague ideas of howcertain data should be .clasSified or categorized.This occurs often with' essay type items. Theclerks can take a portion of the records, note eachreply as it occurs, and mark down a tally for eachtime it is repeated. Soon a pattern emerges andyou can set up sufficient broad categorie0columnheadings) to take care of most of the .answers youare likely to get. Any others can be scored,,in the"other" column. At this point, the researeher isin a position to work up a.code for use in some oneof the other tabulating-techniques- we-will mention'Shortly.
Handsorting This is another counting deViceWhich works quite well with small amounts of data.Instead of rioting the answer andsentering it on yourlally,sheet, you sort the answers into agreed-uponcategories' by stacking up record foims into pilescontaining the same or sufficiently similar answers.Then you count the forms irfeach pile to get ,totalsfor your record sheets, frequency tables, or charts.An alternate approach, where the.record forms arelengthy oij'bulky, is to transfer .the. questions andanswers to special answer cards. Both questionsand answers can be coded or abbreviated to savespace.
The major advantage of the method is its.sim-plicity. You need no special equipment to do thework; and no special skill is re quired to do the-sorting, although there is alv.ays a need for dis-crimination when sorting essay type answers.
A major disadvantage is that you 'need quite alarge working space and relative assurance that`the piles will not be disturbed. Another,is that the
-task becomes quite involved if the record form isbulky. TOrning to the proper page for the sortitem, and then to the page for the-question you areanalyzing, can become irritating arid is very timeConsuming. However, your record form is aone-page,affair, and especially so'if it is on a card,you should hav'e no' difficulty with this phase ofhandsorting.
In other respects, the advantages and disadvan-tages of handsorting are quite similar to..those asso-,dated with the tally sheet approach to tabulation.Both' methods are more subject to error than theother procedures, to' be discussed. On the other.hand, both handsorting and tallying are very prac-tical .and e'dective techniques for determiningappropriate classifications and for preliminaryreporting of seleCted data from..sifrveys'and otherformal studies.
[ 72 ]
Tabulation SheetAn elaboration of the tallysheet. .%1 the answer card as well, is the tabulationsheet. This, consists of a consolidated listieg ofanswers to all usable questions on all records in thestudy. The method is most time consuming hotthere are many opportunities for checking on the\cork. :111(1 volt can use the 1'in:11 product for buildingixleitever tables vou oesire. the tally sheet,you can find errors more eas*. since each (tin:-.t limn:lice is identified on ihe sheet.
The setup is much like a tally sheet. with sitCablocolumn headings and subheadings Likewise. youcan make your entries with t Ypewl her or by-haod.Ilowver. the format differ:, in that \chat volt wouldout iu lhl Margin Or ".titilb- on a tally sheet.is usual-ly It:chided under the major column )ic;o ,icrossthe top. Eah line or "row- across the sh.!et reoortsinformation from zi single reeord form. Fer anillustration of the format 0; :t simpla tabu: 'ionsheet. see the selection entitled, "Evaluating al'astove Improvement Program." appendix A, p,100.
A great deal of thought. must be given to our,Hio ended use of the data before we setup a tabula-Jain sheet. Although we do not have to enter allthe inforMation we have collected. eve should becareful to plan for inclusion of all we are likely toneed for our analysis, Once we have planned theformat and organization of the tabulation sheet,and completed our tabulation. we cannot reline ourdata to include more detail. However. if we allowfur all eventualities, we can consolidate categoric.whene\ er that seems advisable. Forthis reason.we should make our original entries as detailed a:,1.e are likely to have need for them to be 'Whatmay seem like unnecessary extravagance of par-'rand effort it the time may be an et:ono:to; beforethe analyses are completed.
In planning our tabulation sheet. we must alsoconsider all of the possible responses to each ques-tion and plan our columns and headingS according-ly. For instance, where .1 respondent could answer"yes.- "no," or -don't know" to question, orrefuse to answer' it entirely. we might properlyallow a column for each of these four possible typesof responses. This would give us a completeaccounting of all the respondents.
In our example of a tabulation sheet (appendixp. 1001, notice that question .=5 has three pos-
sinle answers: (a) "within past 3 years," (b) "morethan 3 years ago." and it . "don't know"In addition, we allowed one additional 'Idiom] fornonrsponse to the question.
In some instances, One or more Ipac,tions in aschedule may not apply to some of the respondents.In such cases. a column captioned ("Not.Apply " might be added, Referring again to our.example in the appendix. we could quite properly
73
add an -NA- con:inn under question =2. Thiswould then necessitate an entry in the 'NA'.umn for al: I espondents who had not limed !heir
questionnaire =2 and =4 in our thus-ly:01mo.
l'o sepzo ate the data into groups for comparison.IIn records must be classified befor tabulationseots. This indicates the necessity of knowing.oet'ore you start tabulating, which comparisons youwish to make,---what kinds of Tesults you \vitt want.The records can be broken down into any nuMberof subgroups, and the various subgroup totals add-ed together' in various combinations to get variousgroup totals. This is, admittodle, a cumbersomemethod of getting group comparisons and relation-ships, but. it is uffei five and pc ible when eieetriemachine tabulation is impossible.
Machine Tabulation----Unless you are imerestedin (wick answers to a few simple qiiestions, andplan to make no inttinsive study of thy' answers.machine tabulation generally is the preferredmethod of tabulation.
However, this method requlres P.CCOSti to some'cry' expensive machinery that will not be availableto the extension worker for routine evaluations; so'ac will mention only some of he characteristics ofthe method in this manual. In any event; is any ofyou become= involved in in evaluation where thismethod should he used, and facilities can be madeavailable to your research team, you will want toconsult with the operators and supervisors of theinstallation where the data will be processed.
\lost land-grant institutions now have such facili-ties. Extension researchers working on approvedprojects generally can make arrangements to havetheir data machine-processed. Persons responsiblefor the nse of the machines should be contactedearly in-the study.
These people can advise you on setting up yourresearch instruments to secure data most readilyprocessed by machine operations, and they canalert you to the kinds of anai;:_is ,,,-ou can make.They can assist you with coding: problems -and canshow you ways of dealing with free-answer infor-nifitili6ii. :They can suggest ways of using machineoperations to speed up statistical analyses by pro-viding totals ready for processing in calculatingmachines. For that matter, some of the more elab-orate machines can perform basic statistical opera-tions for yon.
Tln immediate costs of machine processing aresometimes quite impressive. however well-justified.For this reason, there should be a clear understand-ing of costs before either the.---researcher or themachine supervisor commits himself beyond recall.They should agree on what operations will be re-quired, and how many. They should agree on thecoding. manner of transcribing to machine record
forms, amount of checki _.r and sequenceof .sorts and tabulation i of c leulations:nid correlations to be do .yo on.
There are various companies making differenttypes of machines for tabulating and summarizinganswers frum record forms or questionnaires. Sonicquestionnaires, set up for certain machines. aremarked with a special kind of pencil so that theforms themselves can be run through the machine.The pencil marks make electric contact for a count-ing of the answers. For other machines, everyanswer on the record form is reduced to a numeri-cal. or alphabetical code; then these codes are punch-ed onto small cards. Coding must be planned bysomeone who thoroughly understands the study.The cards are run through sorting machines whichsorts out the zinswers to any questions we want,giving the number reporting each type of answer.Or the tabulating machine totals the'entries, givingus the sum for all the questionnaires, and the num-ber reporting the answers.
If the questkonnaires, are set up so that theanswers ;. e!1.:!ked are assigned predeterminednumerical va:ues (i.e., precoded), the work and timeinvolved in electric machine tabulation is less thanfor typewriter tabulation. and there is less chancefor human error. The great advantage, however,of electric machine tabulation is the increased free:
.don in manipulation of the cards. Groups do nothave to be all -predetermined. If an iiiie.restingcasual relationship indicated in early summaries`of the data, the cards can be sorted on the machinesimmediately, with no necessity for a completeretabulation as is necessary with typewriter tabu-lation.
Manual Tabulation Using -Punched CardsSys-tems have been developed whereby data are codedand punched on :-ards that are sorted manually.Such a system is useful when the numbei; of 'casesinvolved is limited. Extension-workerscan usuallysecure help and advice on the use of such cardsfrom college statistical units.
Free-Answer TabulationClassification and sum-niarization of answers to open-end que:itions aredifficult. Manv times it is impossible:6 Make ausable classific..;Jon system until the tinswers areat hand so they can be studied as a unit.
If you have a relatively small sample, you canhave all the answers to one question typed, out ona single sheet and determine appi-opriate categoriesby inspection. What generally proves easier, however. is to enter each answer to the question on acard just large enough to hold the informatittri,Then you start grouping related answers and c n-splidating related groups until you fin -ablenumber of categories. Now you can assign a codeto the categories and use it to make entries on
AP
answer cards nr other t:),Jlating devices so thatyou can use these free-answer items as readily asany others in sorting and tabulating.
if you have a large sample; you can save time butstill get the job done by goi-i; through the sameprocess, with a subsample o. your records. Thefirst fifty questionnaires might give you the idea.On the other hand, you might prefer to take every-nth" record.
Quite often, in the course of developing cate-gories, you will end up with an -other" or "miscel-laneous" category. This will accommodate thoseresponses that do not quite fit into your mainclassifications or groupings. This will not detractfrom your analyses unless too many of your rec rds
'other" group. Any time it beeco leswind up in thisyour largest category, you shoulcl ,-onsid.?i- redesign-ing your classification system.
Because so much of the i,Itinre effectivenessof your study depends on how well you handle youropen-end, free-answer, or essay-type questions, it isessential that you make decisions regarding themwith the greatest care. This is an occasion to con-sult with subject-matter and resear pecialists tobe sure that you are making prop dec ions. Youmay evert-Want to set up a vane of experts to\workout a classification wtern for c rtain items.
This difficulty is tfie major rea on why you shouldgenerally restrict the use of ree-answer items.Adequate pretesting should sun st ways to set upcheck-off questions to take their lace: Of course,there will be times when elaborate etesting is notpossible or when you ari, exploring a new topicwhere you have no -idea of what the answers, willbe. In such cases, free-answer questions are a must.
Another problem lies in the fact that check-offquestions limit the range of possible answers. Thismay oontribute to answers which are the result ofsnat: judgment or "any answer to get on with it"reactions. Even. if. you provide an "other" cate-gory, respondents cannot always be counted on tovolunteer information. On the other hand, youcannot be sure how complete your replies to free-answer items really are, largely because of thisreluctance to volunteer information.
In bne case you may have codeable and statis-tically manipulatable data of uncertain validity: nthe other, yeti have had-to-code, hard-to-analyze, butpossibly more valid dal You will have to decide.in ter-is of the particular item. which type of cit!.:s-tiorr...vi:1 pFovide 'Ii: kind of answer that will bemost eticLtive fm your purposes.
Summary
Our concern in t_Its chapter ')as been 1.% h waysof converting maf.txs, of nfouniation into under-standable and usable form tosaid in analysis, report-
[ 74)
8 E'
ing, and decision making. We have suggested thattabulation must occupy an important spot in re-search, and Paluation. and that it is most effectivewhen it is planned at the same time the other re-search instruments are planned.
There are at least four points to consider in plan-ning tabulations: (1) available IcSO,LIMCS, (2yinfor-illation needed, (31 relationships of intereft to 'theplanners of the study. and (4) comparisons desired.We have suggeF4.ed that, when in doubt, it best1(.1 (Ai-ilea data in a more detailed form thi.n youarc likely to use.
We have described some of the common tabula-tion 'techniques, including tallying handsorting.tabulation sheet, machine processing, and manualtabulation using punched cards. The essentialcharacteristics and the relative advantages and dis-advanta sof each method ar,e br !fly discussed.The c pter concludes Nvi.h som( suggestions onthe- andling of free-answer responses.
eferences
Croxton, F'. E., and Cowden, D. J. Applied General Sta-tistics. New York, Prentice-Hall, Inc., 1955. pp. 3749,
[ 75 1
85
Chapter XVI
ANALYSIS AND INTERPRETATIONMI'S. fete( (. Fr SS4
Ward Pf,orti
All of its must work with figures and numbersc:ory day. The simplest assertion in conversation
. deals with quantities .. (Statistical analysis) isone way of bringing -big numbers down to a sin,where they can be grasped with ease."''
What we do with figures once they are collectis the key to the usefulness of research iindings.Without analysis and interpretation, figures havelittle or no meaning. Some degree of statisticalanalysis is within the reach of nvery extensionworker.
1'e-' raps a definition of the terms analysis andinterpretation will be helpful. We can think aanalysis as the ordering of data in such a mannerthat they yield answers to questions. This meanswe take something from our mass of data andorganize it so we can take hold of it. catch on towhat it is trying to tell us, understand it. We breakthe whole of our information down into componentswhich become meaningful to us. In layman terms...analysis is putting data through a mill: we take ourfacts. run them through some sort of statistical pro-cessing, and come up with answers. We put ininformation and take out intelligence.
interpretation, on the other hand, is the processof searching out the meanings of our answers. ofour new intelligence. This process occurs when weget around to answering the "So what?" questions.Interpretation leads to the establi..)inent of explan-atory concepts.
In the normal course of events we would enpectour interpretations to for -fir analysin. It isessential that we have digc lanalyi.ed) data, orknowledge, before we can cnisign meanings and-implications make interPitations). However.it is quite pos(4ible that our original interpretationsmay suggest further needs for analysis of our data.This may be the occasion for a new cycle of sorting,analyzing, and interpreting.
When are plans made for the analysis and inter-pretation of data? This is an area in which muchhelp is needed. A plan for analysis needs to b.well thought out by the investigator when a re-search project is developed. Instruments for col-lecting data must provide tltn information needed
for purposes of an No amount of analysiscan extract from ;ors that are not present
This is not to .,:ever, that we can makefinal and coinput,, for analysis at the begin-ning p;'oject. The is carcher musttin-ays leek for new concepts as observations aremade. nli.oweyer, analysis twist 1-.recede interpreta-tion, even though there is some overlapping.
It is helpful to write ot..t specific plans and toprepare work table in advance. This frequentlyrenuire a 6rea n. deal of titre on the part of theresearcher. It is in.port.ant that administrators..ncierstand tilaf time nd other resources must beprovided for research. In this connection, a calen-dar of work. outlining tie jobs to be done. the timerequired, and respons03llities, is helpful.
thenpnonesses of ana:ysis and interpretation.though of flint-lame:OM importance in the resLar,process, are niten inadequately used by extens;-nnviarkers. is freqn -;ntly due to a limited tinder-standin- f the hasic principles involved. Consni-eralion will now be given to some guiding prin-
/ niples th:n may be helpful to t'ae inexperiencedreseare-ner.
Techniques of Analysis
Statistical analysjs can be enite involved How-ever. all extcvison workers are capable of usingsimple mathecnain2a1 teenniques that will serve thepurpose for muc1- extens;nit studies. These simpletechniques in :une. (1) -numerical counts, (2) per-centages, (3) measures of central tendency or aver-ages. (4) measures of dispersion, and 15) anking.
Numerical CountsFor many of our purposes,it may be sufficient to know how,many. How manyof the respondents knew the agent by name (51 did,17 did not, 2 did not answer)? How many of therespondents were in each age group (5 were under21 years. 37 were 21 through 54 years, 12 were 55years or °rm.)? How many of those who wereshown how to take a soil sample actually took oneon their own land 437 exposed, 13 took sample, 24did not take .ample)?
We need sinli counts to serve as a base for what-ever an;dysis e do. Sometimes this is all we car.
I.76J
8G
to know: at other times this is merely the startingpoint for an involved series of manipulations. Tho.quality of &Lir data, as well as other factors.determine host much more we -are jr: t ifieci in doingwith them by way of analysis;
Caution should be exercised in restricting calcu-lations to simple numerical counts in some cases.other calculations are frequently helpful in avoid-ing (wersimplification or in promoting lirrification.This will become more evident in the discussionsto follow.
PercentagesProbably the next most frequentlyused statistic aft the count is the percent, or theproportion the part is of the wkile. It is a good aidin making relatioosiAps and comparisons mernsing-ful. I-'or example: A relationship like 67 out of 179,compared with 3o out of- 70, is more difficult tograsv, than ,7 perc:mt compared 'with 43 percent.
Another valid reason for using percentages orratios is that they are more easily understood. Thisis particularly true where large numbers arc in-volved. Fur example, 250 out of 780 does not meantoo much to -.'iost people; but 32- percent is mean-ingful.
uug;h the computation of percentages r3to be ri-e.-imple and rather obvious process, the 'c
.several opportunities for the researcher ,J gowrong.
1. Use of the Correct BaseThe base ithe de-nominator or divisor in mathematical terms) is the1111111ber front which the percentage is calculated.Selecting the correct base is important. It is equal-ly important to indicate for your readers or listen-ers what your base is. In a typical evaluation ques-tion. time base could be: 11) the total number ofispondents in the study: 12) ti:.; total number whoanswered the question: or 1:31 the total number towhom the question applied.
In using the total number of cases as a base,roneous conclusions may sometimes be drawn.
'This is particularly true if the proportion of "noresponses" is high. Let us say that out of 100records, 10 did not respond to a certain question.70 answered "yes" and 20 answered "no." Usingthe total of 100 as a base, this means that 70 percentanswered "yes."
U tig the --:anie figures as above, and taking thenumber woo answered the question at all (90) as a
.base. we find that 77 percent of those who answer-ed ihe question reported "yes."
The difference between these two calculations is7 percent. Which is the better answer? We donor know whether the "no responses" would havebeen "yes- "no." ('onseotrently. it is essentialthat the analysis show either that 10 percent didnot answer, or that the 10 "no answers- wen:emitted.
There are many situations in which a question is
I
not applicable to a respondent. Only the numberof persons to whom the particular question appliescan be used as the base for calculating a percenta-age. (',ie example of this is in Table)/
Table I. Broiler practices adopted by 4-H Clubmembers in X county (Number mem-bers raising broilers-75)
Practices adopted
attendbeg meeting(%yhere information vas given)
flowing 1 square footflo space per bird
Using thermostatic controls
E77_I
Keeping 95first week
Number Percent
temperature for
Pretesting brooders 24-48hours r-ior to use
50 100
30 60
20 40
40 "0
45 (10
This table invokes responses to questions askedto determine whether or not 4-11 Club members ina broiler project were _sing recommended prac-tices. In this instance. the county agent is using, asthe base, only Worse who received information onpractices. Till:, drinciple is. often applied when anattempt is being made to determine teaching effec-tiveness.
Another example is the situation where a sched-ul to be used with a cross - section If farmers hasa question on spraying orchards. Obviously, onlythose farmers in the sample Nvho have f_rchards willbe considered as the base for calculating percent-ages.
2. Adding PercentagesBefore we e'en add per-centages, ve must be sure that c:-.,:gories arcmutually exclusive (dO not overlap). For example,let us say we secured the following percentagesw-hen respondents indicated how they receivedinformation about a particular practice:
Source
,radioTelevisionNewspaperNeighborsExtension AgentNot known
Percent131228423815
Even if the percentages had failed to add up tomore than 100, we should have no difficulty inrc4alizing these are not exclusive categories. It is .
possible that'some respondents could have,gainedinformation from more than one source, (:cuiceiv-ably from all of the sources mentioned.
8"1
However, many questions are such that exclusivecategoric:- are possible. This is common among"face" data or background information items. Forexample, the :istribupon of replies to a questionon age might ,-)e shown as follows:
Age group Number Percent
Under 30 years20 through 50 yearsover 7(.1 years
Total
53. 9287 47266 44
C06 400
In thi, case we have selected our age groups sotfril there is no overk, -Ting we are justifiedin adding up our percentages.
3. Averaging PercentagesSometimeperienced researcher falls into e erroring averages of percentagefrequently observed practijustified. The following fi res,,are the percent-ages of 4-H members who iv-npleted projects inthe different districts of a State:
the inex-of report-
Alt , this is a.e, it is never quite
District
A13
C
F
Total
Number Percent
31,0008.000
12,00026,00011,00028,000
116.000
7460(35755072
66 (incorrect)
, the above table. the analyst incorrectly choseto report, as the total percent, an average of thesix district percentages. ITe should have computedthe total rercentage figure by dividing the totalnumber of completions (116,000) by the total Statemembership (165,000) which would result in a fig-ure of approximately 70 percent. Sometimes, as
dle above example, the differences are not great.In other instances, the error can be much larger.
Rounding PercentagesThe researcher mustdeqt167.1Aow many decimal places to include whenhe rep6ts percentages. To show too many digitsmay give a false impression of accuracy: to showtoo few may conceal the fact that differences exist.A safe rule of thumb to employ is to use no morethan are necessary to bring out the differences.
Even when your data are precise enough tojustifyrefined computation of 'percentages, extra or un-needed decimals 'add to the difficulty of readingthe report and they tend to make the figures harderto remember. As a :general compromise, somewriters use nearest whole percentage figures in
[ 78
80
the text of their reports and, wherever necessary,give percentages drawn out to one or more decimalplaces in the supporting tables or is a footnote.
The following three rules cover the rounding ofindividual percentages:
1. Where the dieit(s) to be dropped represent(s)more than half (Le., more than .5, .50, !.2' ) thenumerical distances between the numberabove and the number below, round to thehigher number. For example, the percent-ages 18.6, 18.51, 17.9 when rounded towhole numbs s become 19, 19, and 18,respectiVely.
2. Where the digit(,,z.) to be dropped represent(s)less than half of the distance between twonumbers, round to the lower number. Forexample, the percentages 18.4, 18.49, 17.3when rounded to whole numbersbecome 1S,18, and 17, rt. ,pectively.
3. Where the number to be rounded is exaeqyhalfway between two numbers, round tone1Eiest even number. For example, the .,m-centages 18.5, 18.50, 17.5when rounoedwhole numbersall become 18.
.Jost of the time, ree,,e': occur:: to the right ofthe decimal point. See... °ere:" to the left ofthe decimal point. 7.er-: 4 z s-W)stitu ted foreach dropped (1,.git incomes 170and 16,987 bee .:ines 17,01,0.
Measures of Central Tendency Those deviceswhich help you visualize or identify the centralcharacteristic, the representative unit, or the aver-; are referred to collectively as measures ofcentral tendency. The ones most lik_eiy to be usedby the extension researcher are. (..) the mean, (2)the mode, and (3) the median. Th- are others, butthey require some understanding 01 advanced math-ematics and are not meaningful to larger por-tion of our probable readers or listeners. Conse-quently, they have limited utility in extensionresearch analysis and reprting,
The arithmetic mean, or what most people thinkof when they speak of average, is the most common-ly used measure. To get average income, youdivide the total income reported by the number ofpeople reporting income. To get average age, youdivide the total number of years reported by allrespondents by the number of respondents. To getaverage membership, you divide the total member-ship by the number of clubs.
The advant ge. of using the arithmetic meanare (1) it is most likely to be understood, and (2)it is easy to compute. The disadvantage is that it.gives undue value to fignres at one end or the otherof the distribution. If you were: to average club
memberships of 5, 9, 9, 11, 13. and 37. your arith-metic mean would be 14, which is larger than allbut one of the individual club memberships.
The mode is the answer or value that most fre-quently occurs in a distribution. For example, ifmore farmers report size of fa .n as 120 acres thanany other size, then 120 is the model size of farmin the study area.
Among its advantages are:1. Is probably what most people mean when
they say "on the average" or "typical."2. It is the "most descriptive average in that it
describes the largest number of cases in anyone distribution.
3. It is not affected by extreme values. For ex-.ample, if seven households have the followingnumber of members. respectively-1, 3, 4. 4,
4. 8, 10--the model size family is 4. Regardless ofthe number of persons in the smallest and largesthousehold, the mode would remain the same.
P
Its major disadvantage is that it is significantonly when a large number of values is available.
Another disadvantage is that occasions mayarise when no one group will have numerical super-iority over another. Quite often you come up withtwo groups t`ie saw- size, or nearly so, ;.:!-1 havewhat is called bimodal distribution. When thisoccurs, your amIlysis will be more difficult, but astatistician can tell you how to deal with st;chsitu ations.
The median is the middle value, where half ofthe eases fall below arid half above. To get thisvalue. you arrange ruayl your data from one ex-treme to the other. Then proceed to count halfway- through the list of numbers to identify yourmedian value: In our first example, the arraywould read: 5, 9, 9. 11, 13, 37. The n'imber halfway\ through this array %you'd be our median num-ber. In our case, it would be an imaginary numberas we have an even number of answers. We canresolve this difficulty by saying our median numberlies i.i.jtween 9 and 11, or we can split the differencebetween these -lumbers id call it 10. There is aformula for determining this number precisely, butseldom will we need to he this precise
Advantages of the n-,,-dian are: (1) it Is fairly easyto cale ,late, even, wilt_ a you have to compute theexact r_edian number; and (2) not affected byextereme values or the range of data. Its disadvan-fuges lie in the fact that, (1) it is not..s familiar asthe mean to thelayman. am! (2) making up the arraycan take quite a lot of time.
Measures of DispersionThe 1. :queue), distribu-tion is a classification of the answers or values intovarious groups or classes, arranged in order of sizeor magnitude.
Classifying people according to place of residenceas urban, rural farm, and rural nonfarm is an ex-ample of a frequency distribution.. Such a. distri-bution might he as follows:
Place of residence NuMber Percent
UrbanRural nonfarmRural farm
50 1075 15
175 75
T many cases, measures of central tendency oraverages are inadequate for telling the completestory. This is particularly true where. the spreador range, in the data is great and where extremevalues distort the picture. In these instances, thefrequency distribution of replieswhether on anumerical or a percentage basiscan be a veryuseful technique of analysis. For example, it maybe desirable to know the proportion of young farmfamilies in a county in order to plan programs tomeet their needs.
.
The tte of frequency distributions can be helpfulin checking the representativeness of a sampleagainst census do a. If this is to be done, he classintervals or class( cations satild be ides' ..iii withthose used-for comparison.
In preparing a frequency distribution, 6r::, mustbe-taken to select classes or groups (class i..,.rvalsthat are meaningful in terms of the. local situation.Another requirement is that class intervals shouldnot overlap. In this connection, it would be correctto use "under 10," "10-19," "20-29," etc.. ratherthan "0-10," "10-20," "0-30," 'etc.
There .ire, of course, more complex and precisemeasurements of dispersion than the techniques wehave discussed. For the majority of our extensionstudies, however, nunierical and percentage distri-bution will probably suffice. Whenever we do astudy in which our data have been collected in sucha manner that more precise analyses are justified,we can consult with a trained statistician who willtell us how to use the more advanced methods.
RankingMost people like to know where theystand in relation to their fellows. Clubs lil, toknow ho-W they -compare with other clubs on num-ber of members, attendance and participation rates,and other performance-factors. County residentslike to know where they rank among counties in theState on population. income, percent of farm resi-dents, acres in the different croi, birth and othervital statistics rates, and so on.
Quite often we can use rankings very effectivelyin our evaluation tasks. ,Even if the differencesmay be slight, knowledge that you rank well up ona list can be a source of satisfaction. However, if
o much emphasis is placed on the rankingvaathe,.t an on potential achievements, it can have an
['19]
..rxinhibiting effect on motivation and incentive toimprove.
Rankings are meaningful to most folks, althoughthere is a persistent tendency to interpret rankingsas measurements instead of evidenCes of sequence.It is easy to forget there may 12e only a small frac-tion of a unit difference between' the 30th and 31stranked items and a difference of several units be-tween the first and secon 1. ranked items, or betweenthe last and next to last items.
Rankings can be made up, quite easily if thereare only a few items to rank. You just make anarray of the items ,from, greatest to smallest interms of the units of measurements (inches. pounds,acres, members, completions, adoptidhs). Then youassign consecutive., numbers to the items according,to their positions in the array. You can assign thenumbers from most to least or from least to most,depending on intended use. This pe' makes noreal difference so long as you indicate which wayyou have chosen to assign the rankings.
Sometimes there is difficulty if several itemshave the same size (several 5-member families,many 40-acre farm$, several members with thesame number of project completions, and so on).Frequently, in such cases, it is sufficient to assignthe same ranking to each similar entry.
If you are ranking averages, percentages, orindex numbers of some kind, you may be able toget differences by carrying out your divisions anextra place or more. This is a situation where dif--ferences will be trifling and the resulting rankingsmay have Itle meaning.
If there is no way to break the tie, or if you fe&there is no justification for assigning different rank-
!, ings, you give all equal items the same ranking,Then you must be on the alert to avoid a very con-mon error in ranking. You must leave places inyour. ranking for each tied entry and assign the nextentry the first place beyond these.
If your array reads, G-9, A-8, D-8, H-8, B-6, E-5,C-3, F-3, you woe'r! -^nart rankings, G first; A, Dand H tied s( B fifth; E sixth; and C and Ftied for seventh; not, G first; A, D and 1-I tied forsecond; B third; E fourth; and C and F for fifth,
Rankings are not used a great de;,' is statisticalanalysis beyond simply reporting them, although itis possible to use them to make rough correlationtests. Because of some problems in dealing wit'
rer,it is best to ask a statistician to ex-plain the formula and the limitations of the statisticto you.
InterpretaVon Techn;cp,itls
After we have collected, tabulated and analyzedour data through the various processe's available tous, we still have the task of interpreting the find-
ingS. It is not enough merely to cite our findingsin descriptive fashion. We must ask ourselves thequestion, "What do these data really mean?" Morethan that, we have to come up with the answer tr;,.
the question., or at 'east a statement of what webelieve the findings imply. To guide us in workingout such a ,;t4tement, there are several fundamentalprinciples ...Inch we will now consider.
Generalization-a-Most research motivated by adesire to make inclusive statcients based on alimited number of observations; We desire lo ob-serve results in a limited setting and generalize ourfindings to a larger group or populaion. Thedemands such a desire places on our methodologyhas been discussed at considerable length in otherchapters.
We must exercis caution in drawing conclusionsfor a larger or (different population than the onefrom whi i the sample was drawn. If we collectinformati.,11 from only farm residents in the com-munity, we cannot generalize to the total populationof the community. In fact, if the percentage ofreturns is low. (as in many mail questionnairestudies). or if the refusal rate is high, we may hejustifiek in applying the findings only to the actualgroup of respondents, rather, than to the whole ofthe sample and to the population; from whit' thesample was drawn, as we had intended.
DefinitionWe have no right to generalize forareas or situations where the definitions we haveused in our study do not apply. If ,we would makecomparisons, in terms of concepts or standards orcharacteristics that are not uniformly defined, werftttst at!empt to either adjust our findings so theywill be comparable, or acknowledge that. .compari-sons rcannobe made, or that they are made withreservations and qualifications.
We may want to compare our findings regardingincome in home demonstr, Lion clubs with censusdata. Unless our definitio.: of income is the sameas that of the census, or unless we can go back tothe original data and work out comparable units,we are not justified in making income comparisons.If the comparison is to be valid, incomes must bedefined in the same way.
ObjectivityAgain we must remind ourselves toep personal judgments and other evidences of
s bjectivity'out of our interpretations. Now is thee when the attention we gave to assuring objec-
t vity in setting up our record, forms will pay off.For example, "Did you eat an adequate brpakfast
this morning?" leaves the interpretatioit_ot "ade-quate" to the person who answer the question."What did you eat for breakfast?" equires the list-ing of all foods eaten. The ade acy of the break-fast can then be determined in line with recoin-n,crided nutritional, standards.
80 ]
90
Cause and Effect RelationshipsLOne of thebasic purposes of scientific research is to establishcausality I t he reason whyi. Now we are all awareof, how difficult this is to accomplish in the behavi-oral sciences. however. our scientific in iuisitive-riess compels tis to try to do it. and we feel certainthat as our techniques improve we will becomeincreasingly successful in doing it. The best safe-guard can employ before. Making claims forcause and effect relationships is to screen out allother likely causal factors before drawing our con-clusions.
In the table that follows we have what looks likea good case for the value of extension teaching.This is an illustration from an extension study de-signed to learn the effect of participation in Exten-sion on the rate of adoption of recommendedorchard practices. Strictly on a basis of perctIntagecomparisons between participators and nonpartici-patos. we appear to have confirmation of theeffectiveness of participation in Ext; nsion as a
factor in the adoption of recomi. ndedorchard practices.
Table II. Farm practices carried out-with relationto farm operators' participation in Ex-tension.
Practices adopted'Extent of ParticipationActive 9e active None
Percentage of those withorchards who:
Carried out 3 or morerecommended practices
Applied fertilizerCultivated adequatelyFollowed spray scheduTreated for borersPruned trees,Thinned fruit
However, there were some Other differencesamong the farmers in the sample, in additioe toextent-sof participation. We found that the moreactive participants tended to have higher educationlevels, higher income levels. and larger farm opera-tions. Even though more refined statisticallations might reveal a positLe relationship, there isroom to challenge our unqualified acceptance ofextension teaching as the sole, or at least major,causal factor. It now appears obvious that someof these other factors may be contributing factorsin the adoption of recommended -orchard practices.
'Our point here is that we must be extremelycareful whenever we assign cause and effect rela-tionshps. Before we make such claims we must
be sure to check our relationships against all avail-able evidence. This means drawing or; ,iar "face"data items to make all likely cross-tabulations .toidentify any associations that may supplement -ourbasic relationship in any v,ay. This is also a finalplea to include adequate "face" data and controlitems in all data-collecting forms.
Program Objectives Many of our studies aremade to determine program effectiveness.' Datacollected for this pujpose might be interpreted topoint out how well program objectives' are beingmet. These objectives, as suggested elsewhere, areexpressed most aptly in terms of specific behaviorchanges.
In the table that follows. the desired behaviorchange is constant feeding of phenothiazine andsalt. You will note that close to three out of everyten of-the farms made the desired behavior change,almost a h..lf partially adopted the practice, andabout a fourth failed to act in the desired manner.The effectiveness; of the teaching will be judged inaccordance with what is considered a satisfactoryrate of change for the particular objective.
Table III. "Use.,of phenothiazine and salt on farmsin sample, X county. 1950.
PotentialMuch of our interpretatiori is con-ce ited determining how successful we ha'veb en in reaching our potential clientele. Potentialsa e closely associated with program objectives andwith careful definitions of our population in termsof them. In 4-I-I Club work, for instance, the objec-tive may be to enroll boys and girls who arestudents in rural schools. In this case, the potentialwould be all the boys and girls of 4-H Club age whoare enrolled in rural schools of the area underconsideration.
When it comes to interpreting enrollment figureson a percentage basis, only those boys and girls ofthe proper age and going to rural schools would be-considered.
On the other hand, if all boys and girls of 4-1-1Club age who live in the county arc encouraged toenrog in 4-H Club work, then the total number ofsuch young people in the county -would be thepotential clientele for the action program.. Thisnumber would serve as the base for computing thqpercentage of the potential reached.
Another exainp'' of interpretation in terms ofpotentials relates to pasture improvement. Oneobjective of such a program might be the syste-matic application of lime. In this instance, ourpotential audience is limited to those farms havingpasture land. Secondly, we would consider onlyhose pastures with soil conditions requiring lime.
In other\WOrds, we would not have a true pictureof the effectiveness of our teaching if we includedall pasture acres, and less so if we included allacres in farms.
;
do have a good deal of trend analyi-sis interpretation in extensionlevaluation workiBefore. during and afttr data are oftenthe'..basieof our evaluations of extension teaching iltectivt.::nes,. . Any time we do a benchmark study,we aresetting. the stage for a trend study. Trends arevery helpful in any study of change, whether asindicators of progress or direction of movement.
We\have to be very careful when interpretingchanges, however,'as there are some factors whichcould be involved besides the ones under considera-tion. Frequently, what appears to be a mah p;:edchange may be accounted for 'almost_ entirelya shift in population..
For instance, we haVe the example of the reportby presumably qualified researchers stating thatthere- had been a 10 percent increase in milk con-sumption in their State during a specified period,something like a shift from 2,000,000 pounds to2,200,000 pounds per day in terms of whole milkequivalent. Actually, a realistic interpretation oftheir findings would have indicated virtually nochange in milk congwution per capita during, theperiod. The populationicased from somethinglike 2,000,000 to about 2,2 0,000 persons duringthe same' period; so we can\see that instead ofincreasing, per capita consumption remained almostconstant at approximately one 'ound per day.
We are on even more dangerous ground whenwe try to interpret trend data in -terms of changes
C82]
in dollar income over a period of yeaRs. Inflation-ary ...,ndencies. in economy as a whole mayaccount -for rr -pnarent increase in individualincomes.
When yot. ;elf involved in such inter-pretations .- with a trained researcheror-a sNciai, in order.
-Summary
After some preliminary comments to identifyanalysis as the ordering of data to g A answers toquestions, and in .pretation as the procegs ofsea;ching out meanings of answers, this .chapterpoints out. some principles and techniques ofanalysis and interpretation.
In our discussion of analysis- techniques we referto counts, .)ercentages, mdv.ires of cen,tral ten-denty and dispersion, and ranking: Those tech--niques requiring only an elementary knowledge ofmathematics are emphasized. We also point outsome common errors in the use of each technique.
In kliscussing interpretation techniques, somesafeguards are suggested to assure objectivity andappropriateness in our appraisals. We mentionsome factors limiting how far we can generalize.We stress the importance of agreement on defioi-tions when making comparisons: We point out theneed for adequate_ checking before postulating.cause and,,f'''.ect relationships. Finally, we discussthe importance and value of interpretations in thelight of potentia!S, and trends.
ReferencesI. Croxton, F. E., and Cowden, D. J. Applied General
Statistics. New York, PrenticeIlail, Inc., 1955, Chapters1, 7,8 and 9.
2. Good. C. V.; Scates, D. E. Methods of Research. NewYork, Appleton-Century-Crofts, Inc., 1954.
3. Royal Bank of Canada. "The Uses of Statistics,"Monthly Letter, Jan tlary 1956.
-
Chapter XVII
PREPARING A RESEARCH ORSTUDY REPORT
31 .lowell (;. Fessendritrd F. Porter
If research findings are to be used, they must beorganized and made available in understandabieand appropriate forms. In some cases a completewritten.report is not required. If the results of astudy are to be used only by the investigator, or afew people, there may be no need for a report otherthan analysis tables. Examples of this may includea summary of personnel ratings or a determinationof terests f a specific group with reference tocol erenc seminars and related activities.
n this c e there pay be no distribution of writ-ten materials. In general, however, if researchresults are to be used by many people, a writtenreport will be necessary. 'Sometimes when thestudy is an important one, a single report will notdo justice to the findings. In tihis case, a series ofreports, aimed at. different audiences, may need tobe prepared. The principles of presenting findingswould apply almost equally to both the oral andwritten presentations.
Data art' collected to answer questions. Wheninterpreted, they supply answers to those questions.The report states the questions, outlines the methodof collection and analysis of the data, and explainsthe answers obtained within the limitations of thestudy rbjectives and fincling
In preparing study reports; there are many fact-ors to be considered. This chapter is concernedwith some of the more important considerations.Among these, we will discuss the anticipated aud-ience, ::7,e report outline, contents and organization,writing styles, and methods of presenting studydata. In addition to these, one would also need toconsider available resources, pertinent to preparingand publishing reports.. These 'aspects of reportwriting will be discussed under the three majorheadings of 'planning, style of writing, and ays ofpresenting data.
PLANNING A STUDY REPORT
The Audience
Any writer who hopes to do an 4'ff ctive reportshould have some idea of his prob le audience.
Some understanding of its needs, interests andbackground will help him decide which points tostress in his presentation. With this undergtanding,he will be in a better position to knOW when to slowdown and explain thin /s; 1- t..: will know which detailsto mention briefly and which ones to spell out; hewill know the proper degree of familiarity to usein his writing style.
For our present .purposes, we may conceive ofthree types of audiences (readers or users) for ex-tension study reports: (I),other research people; (2)all professional extension workers-and other educa-tional personnel in related fields; (3) the lay aud-ience. In any one of these categories there may beseveral subclasses. For example, the lay audienceas thought of in this chapter may consist rural,.urban, farm, nonfarm or other people not especiallyconcerned with professional extension work.
If a report is prepared for other research people,it is important to include the following:
1./.Table of contents.2. Summary statement of findings.3. Sp cific purposes and study objectives,4. Significance of the study. This should include
reasons for undertaking the project ,and itsinieaning to the audience concerned'-̀ " "f
Procedures used in the study. This mightinvolve a com-piete discussion of: populationand sampling; methods of data collectionMail, persZmaI interview, telephone, r corn%binations of these methods, record'-f rills used;selection and training of interViel, s; thedates on which field data were coilectec andrate of responses.
6. Star. tical methods and techniques used inthe tabulation, analysis and interpretation ofthe data.
7. Results obtained in relation to the objectivesof the stircly as well as other findi gs of a.
N bYFOCItiCt" itat(qt: ....,---....8. Implications of the findings _in--relation to the
purpose:: of the specific rtt,idy or researc pro-ject. Th3s may Ix- :ess tvrtant for res arch
183]
93
yel-sons than for the other audiences previous -l. -
9. Identification, including the names of peopleresponsible for planning and conducting the.study. date and place of publication, acknowl-edgments.
10. Reproduction cif survey schedules or ques-tionna'..es.
The order in which these phases of the reportare presented is sometimes determined by theauthor. In other instances, the institution or agencysponsoring the study makes the decision.
If the audience is expected to be professionalextension and other 'educational leaders, certainadaptations of the above content, organization andformat of the report may need to be made. Forthis group, details as to statistical procedures,sampling techniques and other methods, used inconducting the study may be omitted' or at leastdiscussed only briefly. It is important to report thefindings in such a way that the extension workercan see their application to his own program: Thismay be the improvement of a method, planningprocedures, or program content.
For the lay audience, a research report needsto be concise, non-technical, easy to read, and maycontain only major findings from the study.,J Rais-ing questions relating to the significance of theresearch for the particular audience may make areport more personal, appealing, and, therefore,meaningful.
The above discussion implies that there may bemore than one report prepared to meet the needsof variotiS audiences. In some cases there may beseveral reports. For example, there may be a very.complete, detailed an ' highly technical report forthe audience of research personnel; a report forprofessional extension workers containing the ma-jor findings and implications, with less emphasis onresearch procedures and t.efhniCal details; and,finally, a relatively simple and brief report for laypeople highlighting the Major findings in a popularstyle.
It is also appropriate to think of a highly compre-hensive study being reported in several specialized"editions." For example, a survey may show cer-tain data on topics of interest to different special-ists.
Resources
Sortie other factors to be considered in planninga research report include r- 'tees available to theresearcher and the or agency involved.Among these are time, finances, facilities and per-sonnel.
Time is of importance from at least two stand-
points; that of time availably() preparre the report.and the urgency in terms of timcliness and use ofthe findings., Adequate time should be made avail-
, able for the preparation of the research report. Itis wasteful of time and money to make an investiga-tion without. adequately reporting the results.
The moncy, lacilities and skills available for writ-ing and illustrating will influence the type, numberand scope of research reports. This should alsobe taken into,account when the research project orevaluation is planned:
It is suggested that the author of a report workvery closely with the editorial staff in charge ofpublications, both in planning and writing a report.Editors can be of invaluable help in selection ofvisuals, improving readability. and effective pre-sentation.
WRITING A STUDY REPORT
The Report Outline
"In planning the report, try to think through thestory that you want to tell your readers; then writethat story in your own words.. Report writing isnot too different from other writing. "_ Effectivewriting is a tool that helps twinsure understandingand use of study findings.
Most writers who contemplate writing a reportfind it helpful to have some sort of ()aline to work,with. It will assure order in the finished work andit will help hold down repetition and guard againstomissions. Such an outline need not be spelled outin complete detail and does not always have to bewritten, but the various points to be included should
re,iven careful thought before actual Writing isstarted.,
Organization and Content
Select only the facts that are important and sig-nificant for ft,. various audiences for whom thereport is being written. It is helpful to arrange,facts in logical order so that they .may be easilyfollowed by th,-.! reader. The use of subheads helpsto call attention to points that need, to be emphasiz-ed and facilitates readability.
Findings should be reported in terms of the studyobjectives. Wherever results are not conclusivesome explanation should be made. It is also impbe-taot that an objective approach be maintained ininterpreting the findings. Where -. the writer isexpressing.a: alue judgment, this should be madeapparent. For example, a writer may state that therate of adoption of a practice was "high" when 25percent of the respondents reported use of thepractice. In reality 25 percent may or may not be
[ 84 1
9'1
a -high" rate. depending on the circumstances orpreviously established standards.
There arc several important considerations re-garding the organization and content of a studyreport.
1. TitleAll reports should start off with a title,..-either on a .separate page or at the ly;al G.' the-
article. Iii the more To.?ular types of repot itshould be short and designed to attract attentionyet give sonic clue to contents; in the more formalreports it can be longer and should giv-: a morepositive clue to contents.
Most of the time the author's name appeal' Ma-. anediately after the title or somewhere on th6 title
ii.page. Sometimes .his name does not appear n themore popular types of report and his age icy ;.designated is the autho.
2. Summary Statement of StudyAh increasing-ly popular practice in report writing is to pacesummary of the report in the fore part of the pub-lication. It may (come befOre or after the table of
'contents-and list of charts. figures and tables, butit always precedes the body of the report. It is notdesignecrto take the place of the chapter on con=elusions which continues to be placed at th'e end.Rather, it pulls together the highlights of the entirereport and 15 resents them so that, (1) the mildly
.interested reader can get the main idea withoutreading further, and 12) the undecided reader canget enough infeH.pation to decide whether or not
...,"...vt-tid the complete report.3. Table of ContentsQuite often the table of
contents is omitted from short reports. and those-of intermediate length which are written for thelay. audience. Any of the longer reports, regardlessof ithe intended audience, need a table of contentsto/speed up reference or to loCate chapters or sec-tions in which the reader may be most interested.In the very, detailed report. the table of contentsanay list subtitles as well as chapter and sectiontitles.
4. Study Objectives and Procedures--Any re-port. whether it is a professional monograph, agraduate student thesis, a journal article or note,a miscellaneous publication, or a leaflet, requiresan introductory section of some sort.
One of the items to be covered in such a sectionis a statement of the objectives or purpose's ofstudy. This statement should also contain the rea-son:, why the objectives were considered important,and the probable use that could be made of thefindings of..the study.
Somewhere in the introductory section thzshould be some comments on how the study wa..made. The amount of space to devote to this willdepend in large part o he intended audience. We
would not expect certain lay people to be interestedin all of the details. although it is a good idea togive them some notion of where the study wasmade, was involved, and-procedures used incollecting the information.
The educators- would want a little more back-ground information to aid them in assessing the
' merits. of the tudy.The details o procedures will be of major inter-
est to other res archers. They will want to knowour population an the sampling plan. They rvillwant to know what\kind of data collecting 'instru-ments we used, and they would to find, a copy,of each in the appendix section. They will'. expectto find some notes' identifying the research teamand what training or 'conditioning was providedfor them. They will be interested in detailS likethe inclusive dates of the "field work, any specialtechniques that were used during the field \yorkphase, the response rate, and soon. .
5. FindingsWhen you have completed yourintroductory section, you start reporting your find-ings, sometimes with the help of tables, ..charts,graphs and figures, or reference to appropriateappendix items.
You should report your results or findings interms of your study objectives and should presentthe evidence both for and against your hypotheses..When you feel you should expreSs a person opinion.you' should say so. Whenever results are not con-ch.Hve, some explanation should be made. Youshould also feel obligated to mention any findingsof a byproduct nature which you consider to becontributions to knowledge or understanding any -'where in the general area of your study. The limi-tations 'of the study should also be pointed out, inall fairness to the reader.
6. ImpliCationsAfter you have presented yourfindings, you have an'obtigation to make some com-ments as to what the findings mean. At this pointit is extremely important to involve subject matterspecialists in interpreting the findings and in sug-gesting implications. Now is the occasion for sug-gesting related topics for future research and forpointing out facets of your own study which shouldbe examined more closely, possibly w ith new data.
The need to include implications in your reportis most urgent when- you are writing for your layaudience. This audience is least likely to recog-nize the possibilities of your findings, partly be-cause of their less thorough knowledge of the sub-ject, but also because of less familiarity with deal-ing in cause and effect relationships.
Most members of your educator audience. willalso needlsomn assistance with implications. Inaddition to your implications, they twill draw theirown after they have considered the report in rela-
[857
95
tion to ken- own experiences. Because they;dothis, you have the additional responsibility of beingcertain you make no implicaticcns which your data`do not justify.
7. Documentation Most reports of research re-quire a certain amouril. of substantiation fromsources other than the research data at hand. Some-tunes, in your report, you may 1,vaht to make corn-Fisons -of your results with those Obtained fromother sources. These other sources should be iden-tified by footnotes or notes at the end of the chapterOr section. All of the sources may be assembled ina bibliography to accompany the report, either aspart of or prior io the appcilix.. .
The bibliography and footnoteS frequently are°Mated from 'reports for the Jay audience but theyshould be in reh,,cts intended for educators andresearchers.
B. Appendix This section serves as t' catchai4for items which had to be omitted from the- bodyof the report for one reason or another but whichwere important enough to be included with thereport. Sometimes side issues are elaborated onin separate appendix mites.. This is where you putyour working tables which are not needed for sup-porting the test but which might be of considerableinterest to some of your readers.
ifere you can elaborate on your, sampling pian orderrionstrate how to deriVe the statistical formulaeyou employed. You can include extra maps, charts,record formS used in data collecting, and any de-vices you 'may have worked up as a result of your
rfindings. .
Although used extensively in scientific researchreporting, appendix sections are not common inpopular reports:
Style of Writing
Effective writing is a tool that helps assure bothunderstanding and use of study findings. Mrs.Cowing, says that the specialist cannot hope towrite "best seller" articles while reporting researchfindings, but he can make his report more read-able. She offers this advice:
"You technical writers cannot assume that layleaders, will be interested in your technical in-formation. You have to identify your technicalsubject with their interests; let them know itwill 'pay them to read it. Slant your writings totheir interests as well as to their needs. Packyour writing with appeal; tell them it's "good-to-eat" as well "good-for-you." Camouflage whatyou think they ought to read with what theywant to read."3Although there arc occasions in extension re-
search reporting when 'professional" writing styles
9 G
are in order, most of the writing we do in Extensionwill be on the "popular"- side. For this reason, wewill devote our remarks to ways of improving ourwriting style to make, it more appealing to oureducators and lay audiences.
Some characteristics of effective popular writingare to: (1)..use the personal approach; (2) use shortsentences whenever poSsible; the sizf.plestword that will carry your rneanit,.g, :rindbe sure a long or technical word understood whenhave to use it (4) confine your remarks to the mainfacts, and keep them flowing in a logical order; (5)hold down the number of qualifying statements,statistical reference's and parenthetical comments;(6) use the simplest illustrations or examples thatwill accomplish your purpose; and (7) use presenttense ..ncl active verbs.
Most of us object to rewriting. but that is thebest 'way to improve -the readability of our writingefforts. If popular writing does not come naturallyto you, just write your report to include all thepoints you want to make, using whatever stv,.. youlike. Then go back over it and br_ea-kc.ip those longparagraphs into shorter ones containing a single
Go through again and-change from the less per-sonal to the ..more'personal pronoUns. Nekt, sub-stitute shorter words for as.many of the long wordsas you can. When ybu must Use technical words,define each one the firSt time it appears. Take asecond look at any sentence over three typewrittenlines king. See if you can make it shorter, or re-write it as several short sentences.
You may have to go back and regroup some ofyour_paragraphs and this may call for some differ-ent transition phrases or sentences. Work in a fewshort subheadings and see if any words or phrasesin sentences need underlining for emphasis. Typeup a clean copy, and see how easy it is to read andunderstand!
METHODS OF PRESENTING DATA
There are several things addition to writingstyle that an author must consider while preparinghis report. None of our audiences would be pleasedwith a report consisting of narrative description'only. They like the presentation to contain morethan the printed word. We will discuss some ofthe more favored ways of presenting research data.
Narrative Description ,
The most common form of presentation, eitheralone or supplemented by other forms, is the narra-tive report Of findings, '"accompanied by whateverconclusions, implications and recommendations theauthor cares to make:--'..The narrative statement
[86 ]
alone can become very boring. but it is appreciatedby th(1SQ «ho do not care to puzzle through tablesor charts.
Tables
We use tables to bring together in rote; and col-urtins larger amounts of data than we would careto describe in words. We are .still at liberty tomake whatever continents we want to about thedata, but we do not have to include so much detailwhen we provide tables containing all relevant data.Large tables, like Table I. generally are assigned tothe appendix section; smaller ones, like the nexttvo,.are more apt to follow along with the narrativedevelopment of the report.
Table I. Percentage of Respondents 14 Years 0.1Jand Over Who !lad, Compieted a Specifi-I!Number of Grades- in School. by Age;Sex. and Industry Group, Upper Monon-gahela Valley, \Vest Virginia, July 1954.
GroupTotal
per,,011;eporl
E1emerslary
0-3Yrs.
Percenlaqe
4.7Yrs.
of
Yrs.
persons:27,;31,:trj.
1.3
ydiose
szltool
4Yrs.
Collorje
1.3Yrs.
4 ormore
No. C.". r
Lqr 1.:
1.111 141. 11
::1 1;.
11 11
1 I.:, 1141 .21
14111 1.; 1
726 11141 1 1
11.41
1.11 1..f
17 1.0 :11
:!71
.ztlore 11
121 7,1
The second table is on'e designed to illustrate afew simple comparisons. It may not involve allrespondents and may not exhaust the informationprovided by a single question. but it presents thedata to support the narrative at the particular pointin the report. It requires a title. headings and afew guidelines. but may not have subheadings orinvolved breakdowns in the stub.
I 37
91
Table II. Ideas Used in Teaching Economy Mealsto Members and Nonmembers
SubjectPercent
Members Nonmembers
Knowing when one food isa better buy than another 80 -19
Kno,wiag what to look forin reading labels 79 49
Knowing how to make substi-tutions in market list with-
-out changing food values 77 35
Meal planning 71
Making a market list 69 35
Using basic seven foods chart 68 40
The third table is a kind which is used quite oftento point up some simple relationship. This kind isincluded in the text, does not carry a number ortitle and usually contains only one column of fig-ures.
Reading and studyingSewingFamily recreational activitiesLaundry work
Graphs and Charts
20'
105-.425
There are many kinds of charts, graphs, maps,sketches and diagi ants that can be used to presentdata. Compared with tabular methods, graphs andCharts have some advantageS as well as limitations.They are more likely to attract attention and im-press the facts upon the minds of the readers. Theyare also more limited as to the quantity of data thatcan ..be presented effectively. A single chart orgraph should not be complicated or attempt topresent too many facts which might -be confusing.Unlike tables, graphic methods can usually showonly approximate values.
Line charts consist of broken or curved lines thzitarc used to picture a time series or trend and fre-quency distributions. Care must be used in select-ing the intervals on both.the horizontal axis (base-line) and the vertical axis. False visual impressionscan be created by using intervals that are eithertoo broad or too narrow for the data in question.Curves or line charts are not appropriate where theitems or units represented are discreet or where itis incorrect to read off values that lie between theintervals on the baseline. In such cases, a bar chartis to be preferred.
.
Bar charts are made up ofThorizontalor-verticalbars and are used mostly for computingtucks. The difference in length of bars on a chartis..for most people. easier to see than. the differencebetween numerical figures. Bars are also usedadvantageously for showing component parts. thatis. the parts of a total.
Pie charts have a wide appeal and are'particularlyeffective in indicating the whole and componentparts. Greater accuracy in reading pie charts, isinsured if numerals are written on the various partsof the graph.
Pictorial charts lend variety and attract attention.However. they should not be used to show too manyideas at once. The main rule to remember is thatan increase in magnitude is indicated by an increasein the number of characters and not by an increaseM the size of the pictures or symbols! Symbolsshould be self-explanatory.
Summary
Research has little value if it is never pulledtogether into some form of report. Although anoral report may suffice for some occasions, wegenerally think of the written report when we thinkof reporting. To guide the author in preparing thereport, we mention several things which can sim-plify and give direction to his efforts.
The first point to consider is the,.probable or in-tended audience. We identify threC major classesof audience, the researcher, educator and lay aud-ience. and give some of the core characteristics ofeach.
The next item we co .si,,kr is the outline andorganization of the report. Its major elements arethe title; summary statement; tables of contents;objectives and procedures; findings; implications;documentation; and appendix.
In our discussion of style in writing the report,we stress the techniques for achieving easier andmore readable manuscripts. Most of these involvesimplicity, brevity and organization, and are mostefficienif,, accomplished by conscientious rewriting
[ 88 1
xvith the objectives of effective popular writingclearly before us.
Our final topic is methods of presenting data. Wesuggest the use of and, briefly describe illustrativemateriajs, such as tables. charts, maps. graphs andpictures.
In the final analysis. decisions on writing styleand methods of presentation must depend on theintended purposes of the report and on its prospec-tive readers. A report must tal the story effec-tively. but this is not enough. There must be apersonal appeal that makes the reader say to him-self. -This report has something in it for me,"
References
1. -Clark. R. M.. and Norford, C. Materials add Techniquesfor Making Charts and Graphs in Teachig Agriculture.Est Lansing, Vocational Education Department, Schoolof Education, Michigan State University, 1954.
2. Cowing. A. G. Let's Write Easy Reading:- U. S. Dept.of Agr., Fed. Ext. Serv. Mimeo, 1111 (11-47), \Washington,D. C., 1947.
3. Cowing. A. G. How to Communicate Ideas in WordsPlan, Write and Trim. Unpublished talk given at theClear Writing Workshop. Leld 1w the Soil ConservationService, U. S. Dept. of Agr., Washington, D. C.. May.1956.
4. Flesch. R. "How to Say It with Statistics." Reprintedfrom Printers' Ink by Division of Extension Researchand Training, Fed. Ext. Ser_ U. S. Dept. of Agr., Mimeo.248 (3-54
5. Good. C. V., and Scates. D. E. Methods of Research,New York. Appleton-Century-Crofts. Inc., 1954.
6. Parton, M. B. Surveys, Polls and Samples. New York,Harper and Bros., 1950.
7.. Strang. R. -Principles of Readability Applied to Re-porting Research." Improving Fducational Research.Washington. D. C., American Educational Research Asso-ciation, 1948.
8. U. S. Dept. of Agriculture, Office of Information. Ex-amples of Charts Use,' in Presenting Statistical Data,N....gative 45145.
9. Yang, E. II. 1).. Fact-Finding with Rural People. NewYork. International Documents Service. FAO. 1955.
. THE USES OF EVALUATION RESULTS
99
Chapter XVIII
APPLYING THE RESULTS OFEVALUATION IN EXTENSION-
,T4eph L. Matthews
Four Objectives
The objectives of this chapter are to help you:1. .Appreciate the potential for .gain that can
come from using results of evaluation in yourextension work.:\ppreciate the importance of evaluation pro-cedures as aids to getting results incorporatedinto action plans.
3. Learn ways of getting results from extensionevaluation ; =-id other steal science research
9
used.4. Be motivated. to use results from extension
evaluation in. your own work.
Three Ways to Improve Practices
We can use three ways to improve our extensioneducational practices. The easiest wny, and theone least likely to provide the best solution, is tomake changes on the basis of our subjective impres-sions of what the problems are. With a minimumof testing. or accumulation of objective evidence.we can make judgments about probable conse-quences or outcomes of our contemplated improve-ments, and be guided accordingly.
Another relatively easy way is to ask a profes-sional research person to study the problem andsuggest solutions. We canoe be criticized for usingthis way on grounds of unreliability of results ordefective methodology, but we can be criticized forpassing up the learning experience of finding outfor ourselves. At any rate, there will be few occa-sions when we can afford to have a professional dothe kind of job that is needed. More than likelywe would have to settle for his stock answers basedon reasonably similar situations reported in theliterature.
Somewhere in the middle ground .between thesetwo methods lies the way which is the subject ofthis manual. Evaluation combines the advantagesof first-hand association inherent in the first way,with some of the more manageable objective andstatistically sound techniques of the latter, into aworkable plan for use by any reasonably competentextension worker.
The final product resembles traditional researchin most fundamental ways, In each there is anattempt to define the problem with precision, toderive the hypotheses from the best available back-ground information, to follOw methods that willprovide facts' rather than subjective impressions,and to generalize from the evidence _with caution.
Basic Differences
The one basic difference centers on the matter ofthe motivation of persons making the study. Thetypical research person is motivated by a desire toarrive at "truth." In his thinking, the degree ofexcellence of the work depends on the degree towhich it contributes to establishing generalizationsthat have wide applicability.
On the other hand, those of us who are involvedin evaluating our own efforts are motivated by adesire to improve ourrawn ways of doing our work.When such.is our objective, coming upwith answersthat have relatively unrestricted application in Ex-tension are of little immediate concern. We wantat 'iwers that apply to our Own particular working
-situation.This kind of purpose does not mean that our
approach to evaluation should be one that restrictsapplication of our fihclings to a particular situation.Quite .the contrary! The sound research methodswe apply while evaluating a particular situationshould produce clues or suggest hypotheses thatcan, in some degree, be useful for analyzing a simi-lar situation in other settings, as well as our own.We owe it to our fellows to do the kind of evaluationjob that will make our findings as meaningful anduseful as possible to them, so long as they firstanswer our own questions.
In addition to fundamental orientation, the work,ing conditions under which the two types of investi-gators carry out their inquiries may differ. appre-ciably. Traditional researchers in education at-tempt to control the study setting so that many ofthe variables normally involved in a real situationare ruled out or controlled in some way by thestudy design. By so doing, they gain precision atthe expense of relevance of findings.
[89]100
Extension workers who engage in evaluation mustconduct their inquiries in the complicated climateof their regular activities. Because of the largenumber of variables in such a setting that cannot becontrolled, the resulting evaluation ;nay he lackingin precision. Neverthelex,. results so arrived athave real meaning becartse they are derived froman inquiry carried out in a real situation. Any solu-tions suggested by such findings would not haveto be translated from the laboratory setting to thereal one.
Disadvantages of Calling inProfessional Researchers
Relying on the professional research person totake care of our evaluation problems has two basiclimitations. We have already suggested the firstone. Such a person cannot *study individual prob-lems- in any strict sense of the word because hisorientation is toward generalization.t ImpersonalHe may be able to think of yourproblem as an opportunity to try out a pilot study,in which event he can personclize the study enoughto provide the kind of answers you need. In mostca,es, however, he must design studies which pro-mise to contribute to general rather than specificknowledge. When most of our problems are pecu-liar to a situation and to the people who are involvedin it. we cannot hope to gain maximum benefit fromthe kind of study he is prepared to make.
GeneralA second limitation is that, even whenthe professional researcher is able to make soundrecommendations, we will have difficulty incorpo-rating Them into our day-to-day activities. It iseasy to.talk about a better way of doing somethingas a result of reading about or being told aboutwhat others say should be done. However, there isa great difference in likelihood of acceptance be-tween change --..commended by others. 'howeverqualified to make recommendations, and changesuggested by our own careful examination of, thesituation.
DetachedThere is a wide gap between talking-change and accomplishing change. The surest wayto bridge the gap is for us to participate actively indecision mak'ng as to what the° change should beso that we thoroughly understand the matter anddo not have to waste time convincing ourselves be-fore we can hope to 'convince others. Then we canproceed to convince others by example as well astalk.
We can expect our associates and clientele toreact much more readily to the suggestion andexample of someone they know personally, such asourselves than to the most positively uttered rec-o ....tions of the professional researcher. Such
a person cannot afford to take the time to identifyproperly with our group. Ile has to rely for rap-port on his professional' title. the prestige of hissponsoring agency, or whatever aura of authoritywe can build up for him:
Getting Results
This suggests the next point we might consider:"How can we go about getting the results fromevaluation used?" A related question is. "Who canuse the findings from evaluation?" Obviously', theonly ones who can use the results of evaluation arethe persons who are involved in the situation thatwas studied or in another situation that is si?nilar.
The next obvious thought is that. regardless ofwho is involved and regardless of the situation, theresults must be made known to the potential usersThe quality of the knowing is critical. Adequatemeans of involvement must be devised if the knowl-edge is to be used to fullest advantage.
We use involvement here in the sense of bringingappropriate persons into intimate contact withresults of the evaluation. Ideally, this is achievedmost fully when the persons in a position to use theresults actually do the evaluation.
The problem is little 'different from gettingfarmers to use the results of scientific research in'agricultural production. The difference. in themain, is in the matter of motivation and the con-sequences of not,.using. When the advantages .are
nnot readily evident, or when the penalties for on-.compliance are not immediately apparent, it isMuch harder to convince anyone that action isnecessary.
Involving Others in Evaluation
As more people become involved in an evaluation,the difficulty of fully involving all of them increasesout of proportion to increase in numbers. Althoughwe cannot hope to keep all of them involved com-pletely all of the time, we can work out ways tokeep as many of them as possible involved as com-pletely as possible. We will discuss things that wecan do to this end in terms of five stages and themain questions to be answered at each., We must realize that some of these suggestionswill not apply for all evaluation efforts, but most ofthem will have quite general application. Manyevaluation efforts are concerned-, with the work ofa single person and may be relat d only to a singlepart of his work. In such evaludt ons, there may belittle or no need to involve others. Nevertheless,there are many occasions when it is best to sharewith others the results of even the, simplest evalua-tions. For this reason, most of the following willapply in some degree to all levels of evaluation.
90]
101
1. When -Deciding to Evaluate Two questionscome to min4 at this stage: "Shall we evaluate?"and "Extictly4whalt shall we evaluate?" Evaluation:Mould be undertaken in response- to an expresseddesire on the part of the potential users of theresults.
Sometimes we may have to take it upon ourselvesto bring the need for evaluation to the attentionof our associates or clientele and to kindle a desirein them to have the evaluation take place. Fre-quently, just showing how to evaluate is enough to
.stimulate a desire to try it. We can ,do this otir-selves or bring in someone from outside"the group'to do it for us. This can lead to interesting themin reading- and reporting on findings of evaluations \or extension research -carried out elsewhere, with \!
the idea of creciding whether or not evaluation isneeded locally.
Once the idea of undertaking an evaluation be-gins receiving serious consideration, the various'members can be kept involved by encouraging themto contribute ideas for the evaluation itself and fOrhow they might be able to make use of 'results intheir own work. This can be done with each
in separate conferences, in meetings Qf theentire group, . and in committee work. rActishould include consideration of possible finding
3. While Evaluation Is in Progress"How are wedoing?" would he the Main question asked duringthis Phase. Folks may not care much about goingout of their way,Kfind out for themselves, but theylike to have sorfie idea of what progress is beingmade_., Whoever has leadership responsibility for.the eauation must see that periodic pi-ogressreporls are made so that interested persons- cankeep in touch, and not have the opportunity toforget all about the matter.
The difficulty 'of reporting satisfactorily increasesdirectly with the size of the group concerned andinversely with the amount of involvement that canbe maintained. The larger the group involved, themore urgent will be the demands on ingenuity fordevising ways of keeping the members informedand- involved. ,Key subordinates will have to assumeever Iargen, responsibilities in this regard as the
aders will have to devote ever larger portions ofTneir time to coordinating and expediting activities,to executing the study plan and to processing thedata..
A reasonable number of fellow extension person-nel and qualified laymen can be- kept intimatelyinvolved with data collecting and processing and:in some instances, with analyzing and interpretingtasks. Others will have to be kept in touch by
and the changes that mightsbe in order as a conse- \.,._.periodie announcements of progreSs so they willquence. Discussion should include possible alterna- not have to be brought back "into the fold" when ,
tive courses of action and their probtible conse- they can make their next big contribution by aidingquences, constructive or otherwise. with word-of-mouth dissemination of results of the
evaluatio-n.2. While Planning. the Evaluation The next
phase is focused by the question, shall wedo the evaluation?" Now is the time to crystallizethinking about extension objectives related to theparticular topic by research, discussion and formalstittements of understanding. This is also an excel-leM occasion for us to remind our associates andclientele of related basic extension objectives andideals.
This is when available alternatives must be con'sidered to decide what kind and qualify of evalua-tion to do. Thorough discussion of methodologyand what can be expected in the way of resultsfroM the various methods will contribute to-betterunderstood decisions regarding budgetary matters.It will also mean better acceptance and more likelyuse of findings when the background is understood.
The entire group can contribute ideaS to be work-ed-into the evaluation plan and to be incorporatedinto the statement of purposes; leaders and fellowextension workers can consolidate the contribu-tions into the final plan. All can voluntder to helpwhen needed with data collecting and processingand, later. to help spread knowledge of the findings.A passive contribution all can make is to supply,without hesitation or reservation, any relevant per-sonal data that may be required for the evaluation.
4. When the Evaluation Is Completed When wehave assembled the information and there are someresults to report, we have reached the fourth stage.
-Now we should be ready to deal with the question,"What did we find-out?" If the evaluation is quiteelaborate, it is a good idea to get out tentative orpreliminary reports on useable information as soonas we can assemble it. This gives- associates anopportunity to get to work on the implications andsome likely applications before interest cools off,and to sustain interest until the final report isready for clistribulio.n.
They can begin work on followup reports andcan prepare reports, news releases and radio ortelevision talks for immediate use. They can tallerreports for the different groups within their aud-iences and can incorporate the material into theirteaching. When the final report is ready, ,theyshould have worked up such a thorough under-standing of the elements of the study that they candevote their full attention to grasping the "bigpicture" as it relates to their work and responsi-bilities,
Insofar as the immediate group is concerned, thefinal report should be something of an anticlimaxIf members have been kept properly informed, they
[91]
should be aware of mostof the contents of the final, draft. Their main use of the document should, be
for future reference oras an aid Oexplaining theevaluation to others. However, some will want touse it .when docUmenting followup reports relatingits findings to those of evaluations clone elsewhere,or to suggest 'areas for future. evaluatiOns. Suchuses are. of course, in addition to the basic one ofpresenting the facts in such a manner that deci-sions regarding change or direction of action canbe made with confidence. In this capacity itsperiod of heavy use will depend on reaction tocontents.
5. Afterward Stage five comes after we havecompleted the evaluation and have made our re,
,,.e.ports. The sort of question we Must now ask Our-selves is, "Who should be informed of our evalua-tion?" Our responsibility now becomes a matterof sharing with others not directly concerned in theoriginal evaluation but who are jn positiohS wherethe information. we have assembled might be usedto advantage:
We should always make it a practice to pass alongour findings, to the State extension training .analresearch persons who will be happy to see that ourinformation comes to the attention of others in theState, or,in other States, who might be interested.If they have not already been involved in the evalu-ation. you may Want to involVe them at this stageby securing their assistance in helping you imple-ment the action suggested by the evaluation.
This is no time to neglect the folks in the studyarea who may have been involved only incidentallyup to this point. They should ha-ye an opportunityto,see the final report to be assured that the eqa1-uation was completed (that the evaluator wasn'tjust doing some busy work) and to get some ideaof what comes next. It will help then gain back-ground for the discussion, 1 ectures, radio andtale-vision talks, demonstrations, and the like tha vill
be coming along in the action. and willhelp them grasp more quickly the intent of the planas a whOle.
Summary
We opened this chapter with a resume of, theti,ree conventional ways of gaining answers to ques-tions. (improve extension educational Practices):Subjective impressions, consultation with profes-sional researcheys, and evaluation. We suggesteddisadvantages of the first two which are resolvedin the last way.
Next, we discussed the difference in orientationor motivation between the evaluator and the profes-sional researcher, indicating that the fornier needs._answers to specific questions and that the lattermust strive for generalizing answers.- We also.suggested that the devices of the evaluator, at leastwhen working on his own problems, help speed upif not eliminate the awareness phase. and bring the ,k
action phase into focus much quicker.We closed with a discussioirof ways to maximi2e
involvement i'n the evaluation as an aid to successfuleventual accomplishment of indicated action. Wesuggested things that can be clone at each of fivestage's (while thinking about the need for evaluation,while preparing the evaluation, while performingthe evaluation, when the evaluation is completed,and while the' indicated action is being carried out)to help smooth out the process.
References
1. Corey, S. M. Action Research to Improve School Prac-tices. New York, Columbia University Press, 1953.
2. Report of Work Group B. Organizing for ... Using ...Research in Esctension. Summary of an Extension Re-searai Workshop, May 6-10, 1957. Washington, D. C.,Federal Extension Service, U. S..Department of Agri-culture.
[ 92
10.J
Chapter XIX
UNDERSTANDING AND USINGEVALUAtION .REPORTS
Joscpli L. 111attliews
Our objectives in this chapter are to encourageextension workers to:
I. Appreciate evaluation' and extension reportsas sources of u..e.ful knowledge.,Acknowledge that a organized approach isnecessary for efriaent use of evaluation andextension research reports.
3. "Learn how to evaluate a report and how toapply the findings reported in it.
.r>
9
Reports Must Communicate
At times we may lose Sight of the real purpose ofevaluation and research reports. -Basically,-a reportis written, by someone to communicate what he,usually the evaluator or research person, thinks isimportant for the reader to know .about the subjecton which he is reporting. Although there are cer-tain things that the writer can do to make his corn-,munication more palatable, there are other things'lying beyond his control that can interfere withhis efforts to Communicate. We will comment onbottfides of. this problem, although the formerhas been dealt with in Chapter XVII.
Inhibitors to Communication
Before going into ways of making -the best useof reports, we might look at some of the factorsthat influence their effectiveness as a 'method ofcommunication. All too frequently, readers ofevaluation or research reports tend to be morecritical of the research methods, and the ways theyare reported, than is justified. This attitude arisesin part from a feeling that the obligation to corn-muoicate is solely that of the writer.,
Obv&r_Ny, this is an unreasonable attitude. Nowr:ter 'can anticipate all the possible situations un-der which the reader might be able to, or want toapply the results of a particulai Study.' The readershould be willing to make some of the modificht:onsor interpretations necessary to relate the study find-ings to his own situation. Unfortunately, this feel-ing is fairly widespread. It interfei-es with muchpotential use of findings from evaltration. and ex-tension research.
Communication in evaluation, more so than insome other situations, must be a two-way proposi-tion. Both the sender and the receiver of the mes-sage being transmitted must be active participatorsin the communication proCess. The writer mustbe able to send clearly and the reader must bewilling to receive the message an then adapt it tohis own use.
Aside from certain inherent limitations of thewritten word, there are definite limitations on what
. the writer can do to make it easy for the reader toget the findings quickly and effortlessly. The pres-sure to simplify involves some danger of misleading.the reader. It is. a rare research person or evalua-tor who is personally familiar enough with all pos-sible applications to apply the findings to allpossible situations. You Must have knov dge of-the practical situation of the moment to makespecific applications.
Likewise, a person who does not fully understandthe Study cannot safely make applications to prac-tical operating situations which differ in any wayfrom the test situation. If the evaluator and theprogram person are-not the same individual, thenthe best substitute is for the evaluator and theprogram person to 31 vork very closely as_ a teamwhile making interpretations and working outpractical applications of findings.
Options and Obligations
When it comes to writing his report, the evaluatorhas three options. First, he can describe the situa-tion in which the Investigation was:done jn suffi-cient detail so that the reader can draw his owninferences about usefulness of findings for anothersituation. A second choice- is to make qualifiedgeneralizations -that still leave ultimate use deci-Sions to the judgment and imagination of-the reader.
The third choice isto descripe specific, situations,either 'actual or hypothetical, in which the findings .
would apply, Then the reader can decide on thedegree to which he considers his own situation tobe comparable to the sample situations, and canmake his own decisions on how widely to apply theevil, ation findings.
[93-]
104
5
This gives the writer quite a range of methodSof presentation. However,- it gives the reactor anequally %vide range of expectations. When presen-tation and expectation patterns are not in agree-ment, the reader may be displeased with the report.This could be an argument for several presentationszis we suggested in Chapter 17, but it could also bethe occasion for a little more tolerance on the partof the reader.
At the same time, limitations of this variety donot excuse the writer from doing his best to writefor the particular readers he has in mind whilewriting the report. This includes the requirementsof writing as clearly and as simply as possible, usingterms and examples that are within the experiencesof the readers, usingtappropriate illustrative andexplanatory, materials. and so on.-. Finally theevaluator should try to give the reader the benefitof his own unique interpretations, which he alonecan wake because of his thorough involvement inthe study.
Personalized Presentations
The written report alone must be regarded as aless than satisfactory method,of disseMinating find-ings of evaluations or res6rch studies. Ideally,as we suggested in the previo4s chapter, the pos-sible users of the results of an evaluation shouldhave the added benefit, of face-to-face- discussionswith the evaludtbr. Fall benefit from any scientificinquiry cannot be achieved without the stimulating,influence of personalized communication processes.Whenever possible, therefore, you should plan tocombine both written and personalized prosenta-tions when you desire to achieve a maximum flowof communication and understanding.
Brevity
Another unrealistic expectation held by sonic isthat brief summaries of findings are adequate sub-stitutes for.full reports. The value of this techniquefor reporting diminishes as the complexity of thebehavior being studied and the number of findingsto be reported increase. Some things a're so involv-ed that brief summaries or 6roati generalizationsare likely to 'be more misleading than informative.
These paragraphs bare intended to demonstratethat written reports' have definite limitations andto suggest are nature of their limitations. The pur-poSe of the remainder of this chapter is to offersome suggestions to help you make good use ofwritten reports in.-spite of any limitations they mayhaw:. The suggestions that follow can aid you in-making systematically effective use of reports onevaluation research.
[ 94
Three Operations
Using reports involves three main operations,each of which has several important steps. Thethree operations are, (1) evaluating the quality ofthe investigation that is eeported, (2) extracting thesignificant findings, and (3) applying the findingsto your own situation.
Your attention may be drawn to a report in anyof several ways. The report may come to yondeskthrough the maqou may come across. it whileseeking information on a related-.topic; someonemay call your attention to it to ask your opinionabout something mentioned in it.
The first 'questions for you JO answer to yourown satisfaction is, "Would this study or ,researchproject be likely to produce results that could havebearing on my work?" AnO'ther question you mustanswer is, "Were the -design and the methods em-ployed in the study sound?" The answer to thisquestion helps answer the next one: "flow muchcon -dence can I have in any findings that arerep Ited?" .
These questions are aimed at something quitedifferent from the repoi-t itself or the interpreta-lions rnade.by the writer. Their purpose is to findout if the study is conducted in a way- that is scien-tifically sound so that only facts are brought outaril so little or no personal bias influences the out-come. No amount of statistical Brecision or carefulinterpretation can correct inaccuracies introducedthrough poor design or faulty methods of obtaininginformation.
4Appraising the Quality-of a Report
Tb do an adequate job of appraising the'qualitYof a report you need satisfactory answers to ques-tions in five main areas. The kind of informationyou need can be summarized under these heaeongs.,
1. Purpost--"INhat did. the investigator try todo?" The important thing here is whether or notthe questions he asked or the hypotheses he testedare ones in which you are interested. If you arenot interested in them, or see no' immediate likeli!hood of being interested in them, there is littlepoint in your going further into the report.
2. Method"What, kind of a study was done?"and "How was it done?" are the qUestions underthis heading. The decision here is whether or not-.the general design arid the techniques employedwere such that valid and reliable information of anobjective nature can be anticipated. The term valid..is used to mean really measuring what was sup-posed- to 'be measured; reliable* is taken to meanthat the same kind of information would be obtain-ed if the methods were repeated on other,samples
105
of the population. As explained earlier, objectivenicp.ns that information Or 'findings tre not influ-t.nced..by the personal point of view of anyone.
a. Population' "Exactly who were. the peoplefrom whom Or about, \.vhotrk information was ortain-
Vott will needlo know this to 'cletehmine how. comparable the test situation is to the.6neto'whichyou desire to apply the: findings. While decidingthe degree to"''.1,-Iiich the. people involved in -thisparticular study 'lie like any group of poople youare concerned i'ith, you will be tryilfg toVind in -.the description-at least some of the same character-istits that are f flind in Your own group.
In most cases you will .find .some cliff,w5ences orthat cerfain characteristics that you (!.(1Sider il)11)01..?ant are not mentioned in the report. In,that event,yop will have to. make personal judgments or esti-mates regarding the characteristics not' mentionedto assign their probable bearing on the findings ofthe study before you can matte use of the findings.
4. Sample='-"What was 'the relationShiP of thesample to the whole populati8n?" and "How was
. the sample cibtained?- are the related questionsasked at this point. The sample must remain amatter of concern, 'even after you are convincedthat the population is quite similar' to your Own: A'poor sample can so distort the reported chal.actelis.-
-tics of the ,population .that the study might as well'be based on '1m-entirely different population.too frequdntly, infOrtua tion 'about sampling. metho-dology-is not included in the publishedreport,,with.,,the result that..rnaw' unjustified assumptions ofcomparability arc made.
.5. Timeliness"How :recently was the,informa-*tion obtained?" and "Does it apply today' ?" Favor-zible answers' to these questions cannot madewithout considering many related factors. A. recentdate of publication or recent-inclusive dates of
r
field woil do. not 'guarantee data which will serveyour purpose. Rapid changes iii conditions of rural
and in technol6gical.knowledge related to-allaspects of contemporary life, force us to weigh the._appropriateness df the findings in terms of detailedaspects pf the individual situation rather than inbroad terms of tempos i1 or geographic similarity.
Date of publicdtion is only a partial check' ontimeline'ss.. The. study may have been 'conipleted-quite some time in the past and delayed montlis'orfears because- of prior publishing' commitments.budgetary problems of the sponsoring agen.cy, or.editorial delays of pile kind Or another. Notinfrequently,. studies are based on research Com-pleted many years previously or on data sources,such as the census, which Thay have.been'assembledten or more yeafs before the time 'they are usedin the...study.
tions in five areas to which you should be. able toranswerJavorably before you should be wit) ing toaccept thd-findings of a, study as appropriate toyour 'needs. :
Our objective in the earlier portions of this chap,-ter -is to make the point that the primary respon-sibility for communicating may be assigned td the
" writer, but that the reader has an obligation to try..to underStand what the .writer is attempting .tocomMunicate. Even when a study, or the reportedinformati,on bout it, may -hae limitations,, theremav, be Many ways that the,findings can serve asa contribution to -knowledge if proper interpreta-tions and allowances are made by the reader:
Summary
ook-Befora Yoti Leap," might well be an alter-nate) title for.thisrehaptor. The major emphasis ofthe last half is- on ways to avoid hasty deCisionsregarding coMparability and assumed applicabilityto yOur' own situation. We have suggested ques-
6L
[ 95 ]
10-G
r),
7,
:77-21
ti
Chapter XX
IN CONCLUSION
Evaluation is a process of systematic appraisalby which k!-e determine the worth, value, or mean-ing of something. This something in Extensionmay be a program or part of one, a method or ap-proach :used in carrying on extension work, or asituation such as a -community, a county, or even alarger area. Evaluation is a process that facilitateseffective decision-making.
Why We Need Evaluation
The ultimate goal, of the evaluation process inExtension is the increased effectiveness of our pro-earns. Evaluation helps in this respect by tellingus how much progress our clienteles. have madetoward their educational objectives. The processprovides us not only with the results of our efforts,exprNsed in terms of Changes in behavior, but alsothe benchmarks by which this progress can be,gauged. In addition, evaluation Contributes to oureducational efforts by giving us necessary inforina-tion about situations for planning sound programs.
In the increasingly complex World in. whichExtension functions, the evaluative process assumes
,growing importance. and significance. E'xtension'sresponsibilities now extend far beyond the -farm_horizons of yesterday. Our clienteles are nowfound in cities and towns, as well as in the opencountry. Our audience is better. educated, bettorinformed, and More sophisticated.
Because of the technological revolution in com-munications and transportation, people can be moreeasily contacted through 'various channels, by massmedia, for example. At the same time people canbe influenced by agencies and organizations otherthan Extension through these same channels.
Because of these. as well as other changes in oursociety, there is grave need for More systematicevaluation in Extension. No longer is it enoughto know that farmers apply fertilizer Now we .should know whether they are using the right kindsand amounts, their attitudes toward this practice.and various other more or less qualitativie items ofinformation. LikeWise, since there are manysources of information cnrrently.available to farm-ers., we must be more zealoUs than ever in deter-
[
lUw
Puiter
mining Extension's contribution to whatever changehas taken place in Our clientele: -
The need to emphasize evaluation in Extensionis evident. The techniquesby.which this process iscarried out are less widely understood. Like anyother skill, evaluation involves a fund of knoN"vledgesand an appreciation of .certain basic proceduresan'd "I:now-how." We hope this manual has madeit clear that any extension 'worker can develop thisknowledge and, with some experience, the neces-sary skill to carry on effective: evaluation. Thishas been our premise in preparing this manual.
Basic Elements of Program Evaluation
Some of the basic techniques and, procedures bywhich surveys and formal evaluations are conductedhave been discussed in the preeedine.chtapters..Expresed in much more summary fashion, we 6nvisualize program evaluation as consisting of fain-procedural .elements: (1) A statement. Of spkificeducational objectives; (2) au inventory of whateverwas done in the name of the program to achievethe objectives; (3); an appraisal of the situationbefore the start ofethe programthat is, benchmarkdetermination; and (4) subseqUent measikernentof the situation to determine progress: and acconi=plishment.. These can be thought of as more or lessconsecutive phases or steps in evaluating any givehprogram.
Pin-Pointing Objectives-.7The first step in.., anyprogram' evaluation calls for an appraisal of specificprograirf- objectives. In this Wstance,f we' mpst,answer the question, "What were we trying toaccomplish ?" The "What" must be Specific enough'to be measurable; and it must spedify the 'changesin the situation (i.e., in the people) that the programhoped to accomplish. It must also indicate theparticular.audience involved.
Program Activities and MeihqdsIn. this,. thesecond phase, lye nitist account for all actions taken:.in the name of the program. This is not always.a -simple task, particularly. where many agencies.iindorganizations are involved. BeCause of 'our faihreto dO this adequately, we in Extension are some-times in the uncomfortable .positiori-of being credit-,.
t .
cd with. or appearing ,to claim credit for, accom--plishnwnts for which 've should. not lOgitimatelyexpect credit
This difficulty can frequently bc), overcome. pro-viding there is close cooperation b6tween both pro-gram pqsonnel' awl the- evaluators.. This. is oneinstance where the right hand must know what the
handis,doiv.
Establishing Benchmarks One of- the mostessential sl.eps, in any program evaluation is anaccurate appraisal of the situation before.e pro-gram -tarts. Unfortunately, this appraisal is Tye-
,. quently tregleeted until sometime after the programis under way. We recommend appropriate bench-mark surveys at the beginning of all. programs. Inan'y event, there his to be some vidence Of whitthe pre-progpm situation.was like bdfore we candecide how flinch pthgtess has been made.
. . ,
Determining Program Progress and Accomplish7ments---The fourth l sic step in evaluation is todetermine progress or end results. Findings or
- 97
conclusions of this nature must obvioUsly be interipreted On-the basis of,p-rogramobjeCtives--i.e,, theanticipated educational changes. They Must alsObe expressed in terms of the pi-e-d4ermined bench-mark, . Further, since weare evaluating extension.work, we' must QonsideT the activities and methods.used pursuing pragrarn objective's:- Only in this;way can we correctly and safely appraise the suc-cess of our eTforts.
The Challenge ,
Evidnationis a tool; it is riot an end in itself: It' sis the means by which extension programs and.-..methcicts can be made more effective. If an evalua-tion is o be effective, the extension workers inMv-,,ed iffitst incorporate the findings into their on -going .
program.All extension workers evaluate; When the
so systematida,tly and objectiMy, they dei,i,/increased personal satisfaction from the sure khowliedge that their work h,. achieved some measureof success:
-.
APPENDIX
109
Appendix A
ILLUSTRATIVE EVALUATION OUTLINEEVALUATING A PASTUREIMPROVEMENT PROGRAM:
Fled P. Prilteh(
ti.`7
The following is an illustrative cutlim. for evaluating theresults of teaching in carrying out part of an extensio:pro ;t am. The main purpose of this evaluation is to findout what degree the teaching on improving pastures hasbeen effective. To what extent were the objectives of theteaching efforts reached?
The decision to be made on the basis of the evaluationwas. ''Should the emphasis on pasture improvement becontinued 1r are pasture improvement practices wellenough established time and effort can he shifted toother pressing problems?" This outline consists of toosteps:1. Background SituationThe need in the county out of
which the pasture improvement program grew.2. Program planningThe procedure by which the prob-
lenr got into the program and plan of work.3. Objectives of the ProgramThe results expected from
. the program.4. Extension Teaching MethodsThis shows what was
taught and ;low.5. Questions to Be Answered by the EvaluationThese
are the specific questions to be answered.6. Sampling the PopulationThis includes the people to
whom the pasture improvement program applies andthe sample.a. Defining the imputation.b. Selecting the sample.
7. How to Get the Iniormi,tionThis step is concernedwith the procedure for obtaining the informationneeded,
8. Preparation of QuestionnaireThis, is the fi)rm onwhich the replies ot those who answer the questionsare recorded.
9. Tabulation, Analysis. and Inter-I.-elation of the Infor-mationInrithis step the informati,!r on the question-naires is analyzed. organized and summarized to deter-mine what it t, elns for the program.
10. Use of the Finding,. The application ot the findings66 program adjustment.
Steps Illustrated
11. Background Situation--The need in the county out ofwhich the pasture improvement program grew. In GreenCounty. many farmers have been changing from row cropsto livestock but their pastures are inadequate. Amongother things the pastures-rieed lime, fertilizer and reseed-ing. The county agent had worked on this with sonicindividual farmers but no concerted effort had been madeover the county.
2. Program PlanningThe procedure by which the prob-lem got into the program and plan of work. This situationwas presented at the annual meeting of the county exten-
siun program planning committee pointing, out that thtabove practices were among the important ones recommended by the experiment station to build better pasture:,After some discussion' the committee selected these pratflees for emphasis in the program. The county extensiolstaff was advised to spend considerable time on this phas'of the program.
3. Objectives of the Program The results expecteifrom the program. From their-knowledge of the situatioiand on the basis of the recommendations of the committeethe county extension staff formulated the following objetLives of the program. These were the changes they hopecto bring about in the farm fa'milies in helping them to hel;themselves.
The objectiveS should be clearly and definitely statedVague Objectives must be analyzed until they arc obseryable evidences and made tangible. They may be whapeople know, do, or their attitudes.
The county extension staff decided upon the followingobjectives of their teaching program.
During the next three years to get at least 400 of the 50Cplivestock farmers to improve their astures..thereby mak
ing it Possible to support more livestock. Four specificobjectives were set down. They were the three practice!stated earlier.and an. additional tracticetesting the pasture soil: to get the I'' estock farmers:
a. to lime their pasturesb. to fertilizer their pasturesc. to reseed their pastures with recommended seedingsd. to test pasture soilThese, then, are not only the objectives of the teaching
program but arc also the results to look for in the evalua-tion.
4.,Extension Teaching MethodsThis shows what wastaught and how. These two columns show what was doneto reach the objectives of the program.. In the secondcolumn are the learning-experiences provided to influencefarm families tovhange.
. [981
110
WHAT WAS TAUGHT
a. Liming and fertilizing would produce a more luxur-ious growth of grass.
b. Reseeding will produce a more complete stand ofgrass and utilize the acreage more Tully.
c. These practices are recommended as the result ofresearch at the experiment station.
d. Some farmers in the county have tried them out withsuccess. richer pastures supporting more livestock..
e. How they can get and apply lime, and fertilizer, theuse of soil tests, and how to get their soil tested.
..
HOW IT WAS TAUGHT
a. Circular letter sent to all livestock farmers in thecounty calling their attention to this situation andinviting them to come to a meeting in their part ofthe county.
h. Meetings in various parts of the county. using visualaids. and testimonies of farmers who had adupte:1 thepractices.
c. Demonstrations on farms in various parts of thecounty.
(1. Pamphlet on improving oastures.e. Timely news articles and circular letters.f. Three major radio talks awl short references from
time to note on the radio.g. Exhibit at the county fair.h. Posters at lime, fertilizer and seed dealers.i. The home demoristration agent mentioned and talked
about these practices at home demonstration clubmeetings. In 4-11 Club work the practices were discussed, particularly in the livestock clubs.
5. Questions to Be Answered by the EvaluationTheseare the specific questions to be answered. The questionsto be answered were. of course, the degree to which theobjectives were reached the results) plus any other per-tinent results that may be expected. These questions: wereformulated as the ones on which information would becollected.
a. How many livestock farmers limed their gestures"h. llow many livestock farmers fertilized their pastures?c. flow many livestock farmers reseeded their pastores?d. many !Restock farmers had their soils tested?e. Ilow many farmers increased their number of live-.
stock?f. What kind of livestock were increased and him many
of each kind?g. What was tip% estimated increase in %aloe of live-
stock?h. Bow did this compare with the cost of the adoption
of the practices?
These and perhaps other questions would he answered bythe evaluation. A questimmaire would he prepared to getanis% rs to the first six questions,- The analysis of thereplies wilulA consider the farms on which the practiceswere adopted for the first time during the three, yearperiod and the livestock increase per acre.
6. Sampling the PopulationThis includes the people towhom the pasture improvement program applies and thesample for the evaluation.
Defining the Popuilation The population was. of course.the 500 livestock farm families in the county. These arcthe people being taught. They are the ones at whom the
1,,priwram was aimed and who are in a position to (10 some-thing about the practices.
Selecting the SampleYou want to get the answers toyour questions as economically as possible least time ;Indeffort) and with as much accuracy as you need. A sampleof 100. chosen at random; from the 500 livestock farmerswill give fairly accurate results and at the same timereduce the amount of time and effort spent on the evalua-tion.
There are several ways of selecting a sample. In thisease every fifth name on the office's alphabetical list oflivestock farmerS in the county was selected for the sample.Selecting, at random does not mean .selecting haphazardly.It means a systematic method without subjective bias.
[ 99 ]
7. How to Get the Information This step is concernedwith the procedure for obtaining the information neededru iinswerifr.; the questions.
The information about the prailices could be obtainedby sending a mailed questionnaire and covering letter tothe sample of l00 livestock families. This looks like theeasiest and mockest way but often is not the case., Manyfamilies don't retain the questionnaire.? without one ormore folluwup letters which extends the time for gettingthe information and increases the follow-up work. You`must have all or practically all of the 100 questionnairesreturned; otherwise your sample may bee..ne biased andthe error increased.
Another method is to interview each of the 100 livestockfamilies in the sample. This method sounds at first glanceto be too cumbersome and timeconstiming. On furtherconsideration it can he made (mite practical. The threeagents in the county could divide up the 100 families; eachagent having :33 or 34 families to interview, taking about
days'or so. The interview not only serves to collect theinformation in a very reliable way; it also serves as ahelpful farm and home visit and as a folloW-up of a pro-gram, thereby showing the interest of the agents whichwould be appreciated by the families.
It could also he done by members of the county programplanning committee after they have been properly trainedto do so. This has certain decided advantages for the typeof evaluation used in this example.
8. Preparation of QuestionnaireThis is the form onwhich the replies of those who answer the questions arerecorded. It may be a questionnaire to mail or for use byinterviewers who visit the farm for the information. Inany case it is a fOrna on which the information is recorded.The example )which follows was partially designed as amail questionnaire.
Questionnaire (Example)
PASTURE PRACTICES IN GREEN COUNTY
Dear Sir:(Date)
a
This is a maned questionnaire sent to 100 livestockfarmers selettec! as a cross section sample in Green Countyto survey the pasture practices used.
Your reply will be greatly appreciated since theability of the survey will depend upon having every ques-tionnaire returned.
We believe the questions can be easily answered in about10 minutes.
We will appreciate your cooperation very much.
Sincerely yours,
(signed by the county agent)
1. Do you lime your pastures? (Check)b. No.
2. If Yes. when did you start to do se?a. During the past 3 years,b. _._ Before 3 years ago.
3. Did yeti do more liming of your pastures during thepast 3 years than you did previously?
a.________Yes: No.4. Do you fertilize your pastures?
b._5. If Yes, when did you start to do so?
a._ During the past 3 years._Before 3 years ago.
111
f3. Did you fertilize your pastures more the past3 years than you did previously?
a.____ ...Yes' .....No.7. 1)o you reseed your pastures? a._____Yes; b. No.8. If Yes, when did you start to do so?
a.. During the past 3 years.Before 3 years ago.
9. have you reseeded your pastures more during the past3 years than you did previously?
a. Yes; b. No.10. Have you had your pasture soils tested during the past
3 years? a b._____N o.11. How many of each kind of livestock do you have on
our farm?
Now(I)
a. Dairy cattle
Threeyears ago
(2)
b. Beef cattle
c. Sheep
d. Bogs
12. How many acres do you have in pasture?a. _Now; b. Three years ago.
13. Have you heard about the pasture improvement pro-gram in the county? a. Yes; b. o
9. Tabulation, Analysis 134:1 Interpretation of the Infor-mationIn this step the intofmation on the questionnairesis analyzed and summarized to determine what it meansfor the program.
When the questionnaires are completed, one or more ofthe agents can edit them, prepare a "typewriter tabulation"sheet and show a secretary how to tabulate the replies.These will he analyzed and summaried by the agents and a.Tort prepared. The following is an example of "type-
writer tabulation" sheet.
Total Questionnaires Received-100
10. Use of the FindingsThe application of the findinto program adjustment. The report will be presentedthe county program planning committee.for its study adiscussion and to determine the extent to which "imprcing pastures" should continue to receive emphasis in ncyear's county extension.program. The results of the evnation will give the agents some clues as. to the effectiness of the teaching methods used.
The information will serve as a topic for a circulletter, radio program or news letter which has certapublic relations values. The agents will have the satfaction of knowing very definitely the degree to whithe objectives of that phase of the program were accoWished and of their contributions in the county. Tfollowing are examples of- raliTs---andsharts.
TABLE FOR REPORT (Example)
Percent of Livestock Farmers UsingImproved Pasture Practices
Pasture Practices
1. Liming
a. Lime pasture nowb. Started liming during past 3 yearse. Did more liming during past 3 years
2. Fertilizing
a. Fertilize pastures now .
b. Started fertilizing during past 3 yearsc. Did more fertilizing during past 3 years
TABULATION SHEET (Examp e)
PercentLivesto.Farmer
80
60
70
75
70
75
County
Question-naire numberand/or name
1. Limepasture
2. Startedliming
3. More limingduring past3 years
4. Fertilizepasture
5. Started fertilizing
a. Yes b. No NR a. Past3 yrs.
b. Before NR a. Yes b. No NR a. Yes b. No NR a. Withinpast 3
b. Morethan
Don'tknow
NP
Years 3 yrs.ago
I X x x X x
2 X X X
3
4
5
X
x
X
X
x
X
x
X
x
x
X
x
This il ustrates the tabulating procedure. Count all thO X's in column to and write that number at the bottom of column 1a.
.- Iota/ 10
b.
15 5 70 20 10 70 20 10 75 '4 0 50 15 15 10
I. NR means "no response"
[ 100]
112
CHARTS FOR MEETING OF COUNTY EXTENSIONPROGRAM PLANNING COMMITTEE (Example)
PasturePractices
Liming
Fertilizing
10%
757;.
Percent of Livestock Farmers UsingPasture Improvement PracticesMore Since Program Committee'sDecision Three Years Ago.
Percent
20 40 60 80 100
V71 41/ A1/261
Ai/ .059:401110:
PasturePractices
Liming
Percent of Livestock Farmers UsingPasture Improv.ement PracticesBefore and After Program Commit-tee's Decision Three Years Ago.
Percent
0 20 40 60 80 100
Total I
Before 20% After 60%80%/ OSZE1=?Z2Z2Z2
Before 5% After 70`:,
Fertilizing 75% bi=227Z22==22
Appendix B
EXAMPLE OF AN INFORMALEVALUATION
Mrs. Laurel K. Sabrosto
This example of an informal evaluation illustrates theimportance of planning ahead of time and involving learn-ers to assure fullest interest and maximum benefits.
Evaluating a Series of InService Training Meetings
The audience at these meetings was about 30 or 40 exten-sion workers: the speakers and discussiOn leaders wereauthorities in the various subject-matter areas: the plan-ners i.vere extension workers with training responsibilities.This is the procedure that was followed:
1. Each speaker and discussion lea(!er was asked to pro-rid- the planners. before the meetings started, with astatement of the educational objectives he had for hits partof the prograni. in terms of what he .expected the learners(extension workers to get out of it. The, speakers anddiscussion leaders were completely cooperative., in provid-ing this information.
2. Near the end of the series of meetings. the audiencewas formed into working groups, and each group discussedand decided on the most important things it -had learnedfrom each talk or discussion.
3. The evaluation discussion leader placed. on, a black-.board the things learned by the audience from each talk
L102
114
or discussion. He placed these on one half of the black-board. Then he wrote on the other half of the board theexpectations of the speaker, or discussion leader.
. The total audience then reviewed the similarities anddifferences between what the speaker or discussion leaderexpected and what they thought they had learned. Allspeakers and discussion leaders who were not involved inother professional chilies at the time were present for thisevaluation session.
5. Members of tlic audience discussed what they wereexpected to learn, why they kid learned what they wereexpected to learn from some speakers and discussion lead-ers, and why they did not learn the expected things fromothers. The evaluation leader brought out that they were
.following the approved method of evatuating an educa-tional effort: (1) Doing so in terms of the objectives of the-teacher," (2) involvjng the ,"learners:' (3) analyzing theresults to see reasons for success and failure, and () look-ing ahead to, occasions for applying the results to similarsituations.
This evaluation procedure not only illustrates a meansof evaluating properly, but also providiks a method ofreemphasizing the points the audience has learned, and ofbringing to them points that they have missed.
Appendix C
PROFESSIONAL HELP FOR THEEVALUATOR
Iw,t ph I, . Matthews
The potential area from which to select an evaluationproject in extension teaching is as broad as Extensionitself. Any project you might select could deal with anyone phase, or a, complex of phases, which could rangefrom one extreme of extension activity to another. At thesame time, no extension worker can hope to he an author-ity in all areas of the work. Therefore. any of us who wishto pause and assemble information that will help us im-prove our work need have no feeling of' embarrassmentover not having all necessary knowledge and skills tofollow through with appropriate evaluation.
For example, many of us are trained in special areas ofagriculture or home economics. Once we are on the jcib,however, we find that many of our administrative andcoordinating tasks require considerable understanding ofthe social sciences. We need to know techniques foridentifying differences between persons who should orcould be reached and those who actually are reached byour teaching methods. We need to know why certaingroups among our clientele do not respond so well to ourprograms as we believe they -should. We need to knowthe differences between urban and suburban families inour county, and how these differences may influencetheir demands on our services.
We need to know how best to approach certain familiesor individuals how to work through their needs. values,motivations, ambitions, knowledge and understanding. Weneed to know which methods will work most effectivelywith which groupshow to approach Them how to makethem interested, how to hold their attention, how to en.courage them to follow Lhrough to. completion, and howto carry on by themselves.'
Although these are everyday concerns with most us,it is easy to see that gaining the necessary information inthe proper form to make the kind of evaluation that wouldhelp us in our work would require more than routineunderstanding of evaluation methods and techniqiies. Inspite of the certain value such knowledg:: would have forus, the size of the project makes us pause to consider thequestion, "Should I undertake evaluation activities thatare beyond my present competence ?"
Such a question implies that we feel we haye to "go iton our awn" in evaluation. Actually, there are many pro-fessional resources which arc at our disposal. The kindof question we should he asking ourselves is, 'Are theneeded resources and technical competences available tohelp with the evaluation I want to make?"
When we state the question in this manner, it is mucheasier to gain the confidence to start for, in most instances,we will find that all the professional help we will requireis at our disposal. All we have to do is look around andtake counsel where needed from persons who are'compe-
`tent in the areas that are difficult for us or unfamiliar tous. With their help, we can move on to appraising whatresources we will need and finding out whether or notthey are available.
The final decision to undertake an evaluation, and how
extensive the investigation shall be, will depend upon theavailability of technical help and other special resourcesthat must be marshalled for the job. Technical help isessentially a personnel problema matter of contactingthe right people and enlisting-their knowhow. The otherresources consist of time allocations by persons involvedin the evaluatiorl. and any special materials or facilitiesthat may have to je assembled.
As in all investigative activity, the evaluation projectfinally outlined and executed will be a ; ompromise be-tween what we would like to do and what we were able to
. do. We find out sooner or later that others are not soconvinced as we are as to the relative importance of theevaluation we contemplate and that they have other ideasfor using limited available common resources.
Before we complete the evaluation task we can expectto make numerous compromises for reasons of economicor practical expediency. However, compromises providepractical educational experiences in resourcefulness andpersevercnce and should not be taken as reason for drop-ping a research er evaluation project unless they invalidatethe basic objectives of our efforts.
If you have never been involved in a systematic evalua-tion activity, you may be at a loss as to where to turn forthe help you need. This confusion should not last long,however, as we in Extension are fortunate in that we haveready access to a wide variety of professional guidanceand help.
All of us have access to the technital and professionalresources of our own land-grant institution, and some ofthese resources are exceptionally helpful. Of course,.there will be some variation in the particular assortmentof competences represented in a particular State extensionstaff, resident teaching staff. or experiment station staff,but you can count on finding someone who can help youget a start, or who can tell you where to look if he doesnot feel qualified to advise you,
Availability of such persons may be limited by otherdemands placed on their time, but most of them can findtime to get you started with your evaluation. Anotherfactor_which may interfere with their effectiveness insofaras your work is concerned is the distance from the institu-tiGn to your study area. If you can run in or your con-sultr,i can come out in a few minutes, you can get togetherabout as often as you care to If the trip is one whichwill require hours or d9ys, both of you will have to planyour meetings carefully, and work up agendas for whenyou can get together.) At other times you will have tor.resort to correspond ce or to working with substituteshe may be, able to r ommend.
Many times he can suggest specialists and facilities ofhis institution which are located off-campus but which maybe more convenient to your study area. There are manyfacilities of this nature which you can feel free to drawupon. No doubt you will be amazed when you learn howmany of them there really are. For the balance of this
[ 103
appendix section, however, we will .confine our remarksto facilities available from the landgrant institutions andfhe Federal Extension Service.
State Extension Staff
You are no doubt aware of the many specialists on theCooperative Extension Service staff of your State but youmay not be aware that over 20 States have a specialist inextension research on the staff. Your State may be oneof them. This staff member is the proper initial contactfor aid in any kind of evaluation project. In mast instanceshe has had special training in the techniques of evaluationand is quite likely,to be highly competent in social scienceresearch.
lle can assist you personally in planning and followingthrough on an evaluation activity. In addition to the helphe can give you personally, he is in touch with otherindividuals in resident research and teaching who havethe skills you may find it necessary to draw upon.
Most of the 30odd State personnel training specialistshave some competence in evaluation and research in gen-eral. In the absence of a State extension research person,the leader of 'the training program is a !,00rt person tocontact.
Most State staffs have e,ne 9r more oersonS who haveparticipated in formal evaluation work, or who have take1the evaluation course at a regional extension summerschooi: if there is no extension research or training per-son on the staff. a person with this kind of backgroundcan he consulted.
Resident Teaching Staff
All resident teaching staffs at the land-grant institutionshave individuals with backgrounds of research in educa-tion or the social sciences. Usually there is someone onthe staff who has had research experience directly relatedto almost any conceivable evaluation problem we mighthave. All you have to do is find the proper person andmake suitable arrangements to get the help you needfrom him.
Good persons to seek out when you need help on evalua-tion are the ones teaching research courses ox thoseengaged in research work in the various subject-matterdepartments. The more experienced staff members indepartments of mathematics or statistics, psychology, edu-cation, economics, sociology and anthropology can makehelpful suggestions regarding evaluation methodology,Although their suggestions would have to be modified tofit your particular needs, they should be helpful in extend-ing your understanding of general approaches and basicinvestigative techniques.
Experiment Station Staff
Every experiment station has on the staff persons whosework is planning and following through on a wide varietyof research projects that involve esser daily the same prob-lems of design and execution that .ve encounter in ourevaluation projects, This is particularly true of the peopleengaged in sociological research, sampling methodology,questionnaire design, and data collecting and processing.
Most experiment stations have specialists who devotemost of their time to working out coding plans for ques-tionnaires and schedules for either hand or machine tabu-lation. They also serve as consultants on questionnaire andschedule designs. Usually there is someo.le who is experi-enced at preparing tabulation plans in sufficient detail sothat the followup work can be done by clerks or machine
operators. All of them are available to advise you in pro -cessing your own data for analysis and interpretation.
Most of the experiment stations have modern electricalind'ehines for tabulation and other analysis operations tospeed up data processing. In many .States the facilitiesare adequate to handle almost any statistical job likely tobe undertaken at the institution. During slow 'periorISthey are made available to outside researchers. Atysiichtimes you may he.able to arrange for use. of the machines,together with the services of the operators, to' processyour data.
Adequacy and availability of facilities and qualificationsof staff. memliers and technicians will vary fr?m one insti-tution to another, but all of them will be able to help agreat deal. Most of the time the resources are more thanadequate for routine evaluation work. This is, particularlytrue of the kind of evaluation we would do by ourselvesor with the cooperation and assistance of a few colleagues.
Even overloaded resident teaching and experiment sta-tion personnel are willing to advise us to the best of theirability .when approaeh'them through prober channels.When they are convinced we are 4ncer6 in our efforts atevaluation, they will be most tolerant and understandingof our comparative incompetence, and will go out of theirway .and make personal sacrifices to be helpful. .
Division of Extension Research and Training
This Division ,f the Federal Extension Service is aservice unit that has as one major responsibility, thenational leadership in 'extension research and the stafftraining in evaluation. This combined function is accom-plished by coordinating extension research and, evaluation,by training and assisting State research persons with theirwork. by providing courses and training sessions in eval-uation, and by consulting and participating directly inevaluation activities at all levels.
The size of the' staff and the facilities available to itlimit active participation in evaluation activities to a re-stricted number of major 'national, regional, State andcounty studies. Most of these are clone as cooperativeprojects between the State or regional group and theDivision. All of them must be of the pioneer variety tosome degree, although a certain amount of replicationoccurs.
The major contribution of staff members is madethrough personal consultation in the States or throughcortespondence with individuals to advise ^them on eval-uation activities and extension research projects. Assist-ance provided may consist of giving suggestions aboutsources of needed %help, supplying materials, advising onspecific phases of the work, or reviewing plans for allstages of the contemplated evaluation sequence. .
Requests for assistance from Division personnel by.workers in a State should be matte through the State Direc-tor to the Director of the Division of Extension Researchand Training or to a particular staff memeler of the Divi-sion. If there is some question as to the identity. of theperson who can give the desired help, the request should
be made to the Director who will refer it to the proper staffmember. Otherwise, the State Director can communicatedirectly with the proper staff member.
The general categories of help that can be given by theDivision are listed as major topics in the accompanyingoutline. Under each major topic are listed the specifickinds of assistance that are, offered in that area.I. Defining the problem and determining necessary re-
sources.a. Spelling out the problem and the general nature of
the information that will be needed.
[104]
nc
h. Investigating resource needs and possible Ways ofmeeting theni.
c. Analyzing tiie time requirements and preparing apreliminary calendar of work.
d. Analyzing the nature and extent of the: cooperationthat will he needed from other extension workers.
2. Desigring thc evaluation plan.a. S:ielling_ out what was taught that is to be evaluated.b. Deiiiiing *hat aspects of the teaching action are to
be evaluated.c. Deciding upon objectives of the evaluation.d. Deciding what evidence is to be collected.
3. Preparing research_forms.a.ADeeiding what questions to ask and how to phrase
them.h. Developing the record format.c. Preparing instructions for handling the questions.
4. Planning tabulations and the analysis.a. Preparing the coding plan.b. Deciding on the tabulation method.c. Preparing tabulation forms.d. Deciding on statistical techniques.
5. Preparing the sampling plan.a. Defining the population.h. Determining the sampling fraction.c. Preparing the sampling instructions.
6. Conducting the field survey.a. Selecting interviewers or recorders.h. Training interviewers or recorders.
V
1 105 ]
c. Sopervisin.; field party.d. Editing srhedules.
7. Preparing the report.a. Planning the-report outline.b. Analyzing fhe data and drawing implications from
them.c. Writing the report.d. Publishing the report.
8. Getting findings disseminated- and used.a. Interpreting the findings.b. Drawing implications for action programs.c. Planring the disseminating activities.d. Setting up reporting situations.
In addition to the direct help that members of theDivision staff can give, there are 'numerous duplicatedmaterials on evalilation in Extension that arc available fordistribution. These materials consist of examples of ques-tions or complete questionnaires, various kinds of evalua-tion check ,sheets, tabular layouts and recording devices,and brief statements on methodology for the differentphases of evaluation. They also have samples of differentkinds of complaecl evaluation reports.
Staff members are prepared to answer most routinerequests for suggestions about how to handle specificevaluation problems. In many cases they can provideappropriate example materials to help make their sugges-tions clear. When the problem is a new one to them, theywill do all in their powLr to work out a practical solution.
Appendix D
EVALUATION REFERENCES INEXTENSION SERVICE REVIEW
From time to time the Extension Service Review containsarticles on the need for evaluation, planning evaluation.methods of evaluation results of evaluation, and uses ofthe results. Some of these references arc listed below.
January, 19551. Gallup, Gladys and Matthews, J. L.-Appraising
-Farm and Home Unit Approach. p. 19
April. 19552. Cowlitz County, Wash.-Who Are the Members of
Home Demonstration Clubs? p. 84
May, 19553. Sabrosky, Laurel K.-Some Ways to Hold Your
Young People. p. 1014. Clayton. Leon 0.--- Factors for Successful 4i Club
Work. p. 105
September: 1955,5. Matthews, J. L.-What Is Evaluation? p. 1716. Fifield, Marguerite and WtIitrnan. George E.-After
We Interviewed 346 Families We Revamped thoCounty Program. p. 173 ,
7. Knutson, Andie L.-How Do You Measure Progress?(p. 174
'' 8. Procter, Harriet-Ou- Time-Use Study Was an EyeOpener. p. 176
9. Harris, Margaret E.-We Take A Good Look at theWomen We Serve. p. 177
10: Fessenden Jewell-Tools for. Testing. p. -17811. Raudabaugh. J. Neil-Program, Plan of Work, Pro-
gram Evaluation. p. 180'12. Ueland. Gale-Half of Louisville Listened_ p. 18313. Clark. Robert C.-The Home-Focus of Our Re-
search. p. 18414. Frazier, Leslie-Our Families Wanted-Facts. p. 18715. Schruben, Luk, M.-Compounding State Research.
p. 189
February195616. Pierson. Arlie A.-One Way to Measure the Value of
Mass Media. p. 45
March, 195617. Blanchard, Evelyn and Shipley, Fern-Four State
Staf: Pool Ideas. -tor a Revised 4-H Food Program.p. 55
May,' 195618. Copp, James H.-Why Some Stay In band Others
Drop Out of 4-H Clubs. p. 9719. Kern, K. Robert-Iowa Folks Fou.'d Hy: Answer to
Facts-So What?,p. 10020. Marchant. John D. and Rowe, M. Pat,Lne-Our Time-
Use Study. p. 103
June,21.
1956FES Task Force-Earmarks of Farm aud HomeDevelopment. p. 109
July, 195622. Dewar. Carmen L.-Iowa Women Learn to Serve
Nutritious Meals. p. 13223. Crosson, Mary Alice-Indiana Specialists Test Their
September, 1956 r25. Ross, John E.-More Than Brick and Mortar. p. 15726. Longsdorf, Lisle L.-Facts and Dreams. p. 16327. Several Counties-Let's Tackle One Problem at a
Time p. 16628. Wyoming Extension Specialists-Analysis of Needs
-A First Step in Program Protection. p. 17029. Swanson, Harold B.-Early Results of Program Pro-
jection. p. 172
November, 195630. Pope, Edward V.-Do You Know Yourself? p. 20431. Beers, Howard W. and Rheinwald, Hans-Evaluation
Takes Root in Eurqpe. p. 2063Z. Tlchenor, Philip--A Hazard Hunt. p. 207
December, 195633. Dodge, W. A. and Bradfield, Alex-Vermonters Im-
prove Milk Flavor to Increase Sales. p. 22134. Stanislaus County, California-What Extension Help
Do Women Want? p. 235
January, 195735. Erickson, Anna Jim-Listen to the People Speak.
p. 11
March, 195736: Bergen County, New Jersey-Food--No. 1 in 180,000
Budgets. p. 63
April, 195737. Reeder, Robert-We Took Another Look. 9. 7738. Liveright, A. A.-What Makes Them Tick-My
Observations 'of Two Successful Community organi-zations. p. 79
May, 195739. Rutledge, Lloyd L.-Clues to 4-ll Club Program
Vitality. p. 10540. Dildine,. Glenn C.-Your citizenship Improvement
Study. p. 107Sabrosky, Laurel K.-Let's Be Challenged. p. 108
July, 1957"41. Holloway, Margaret E.-We Used the Facts We
Learned From 500 Interviews. "p. 13942,. Straus, Murray A.-What We Can Do for the Part-
time Farmers. p. 153
[106]
11G
October, 195743. Welch. Jr., C. HermanDaily Records: Chore or
Rich Harvest.? p. 20344. Barlow. ShirleyThe Office 'SecretaryEnvoy of
Good Will. p. 20645. Welch, Jr., C. HermanKnow Your County People.
p.210
December, 195746. Sabrosky. Laurel N.Evaluation as Related to Public
_Relations. p.25'2
January, 195847. Leagans. J. PaulGive Your Fitress a Checkup. p. 6
March. 195848, Porter. Ward F.----Studying Our Human Resources.
p. 57
1' 107
April, 195849. Hill. Kate AdeleThree-Way Approach Pays M.
p. 79
May: 195850. Carter, GlennWants and Needs of Older Youth.
p. 108
September, 195851. Shipley, FeinProbing to Determine Needs.. p. 1,8-2
November, 195852. Sarbaugh,` L.-E.Have You Tested Your. Audience?
p. 219.
January, 195953. Frutchey. Fred P.To Get . . . or Not to Zlet
