continued on page 3

Agriculture ResearchProgram to beEvaluatedExcerpted from an interview with Dr.Fleming, NIOSH Office of ExtramuralPrograms

Systematic program evaluation isthe new business model for gov-

ernment that has evolved from GIP-PRA and the OMB ProgramAssessment Rating Tool (PART)http://www.whitehouse.gov/omb/part/2004_program_eval.pdf .NIOSH is committed to using anexternal expert review, one of themost valid and accepted evaluationmethods, to assess the effectivenessof its research programs. TheNational Academies (NA) has beenengaged to evaluate NIOSH’sresearch programs with respect totheir impact and relevance. Thisexceeds what the agency is requiredto do. The Agriculture Initiative isone of fifteen NIOSH program areasthat will be evaluated. Dr. RoyFleming, Assistant Director forAnalysis, Translation & Operations inthe NIOSH Office of ExternalPrograms, is working with the nineagriculture research centers and theNational Children’s Center to pre-pare the evaluation package, whichwill also include other grant awardsand NIOSH intramural activities forthe period 1990 - 2005. Using alogic model to identify resources,activities, outputs and outcomes,each Center will produce a report thatdescribes the overall “program” value ofthe center from its inception to the pres-ent. The external review is slated tobegin in August or Sept. 2006. Seehttp://www.cdc.gov/niosh/nasfor details.

Agriculture has advanced in technology, but itremains a high-risk sector for occupationalinjuries, illnesses and fatalities. Deliberately

structured intervention research pro-grams have addressed agriculturalsafety and health and are being evalu-ated. For example, an AgriculturalCenter Evaluation team (ACE) isworking to develop and apply consis-tent collaborative evaluation meth-ods. In 2006, The NationalAcademies of Science will reviewand evaluate agricultural projectsfunded by the National Institute forOccupational Safety and Health.

The science of evaluation in publichealth has proliferated since the con-tributions of William Haddon whoauthored a landmark book on“Accident Research”. The “HaddonMatrix” employing “two by two”tables, has become a widely accepted conceptual eval-uation tool. One of Haddon’s associates, EdwardSuchman, produced a classic book “EvaluativeResearch” in 1967. A major contribution of theseauthors was to place the concept of “accident-prone-ness” into perspective, and to establish that injuries, aswell as illnesses, had associated causal or risk factorsthat could be characterized.

Public health increasingly employed evaluationmethods after 1970. Many external developmentalaid programs in health and agriculture mandated eval-uation of effectiveness, for example, health promotionprograms in the Caribbean. In the US a team pro-

duced a “Manual for Self-evaluation of ProgramsServing School-age Parents”. The team learned thatmany such programs lacked coherent, defined, attain-

able, measurable goals and objectives,even though they had been in operationfor some time. The diverse staff mem-bers providing specialized services tomeet the health, educational, financial,and social needs of clients tended tosee their own specialty contributions astop priority. This confusion interferedwith cohesive team function. Attemptsto evaluate program effectiveness andprogress often were frustrating andinconclusive. Consequences of evalua-tions were sometimes destructive.Currently, evaluation results are moreoften and more properly used to pro-mote program improvements.

Similar difficulties still arise in eval-uating action-based research if there is

a lack of up-front clarity about the purpose and expec-tations. Difficulties also occur if expectations changeover time. Evaluation of work designed to producedescriptive data or feasibility results should be limitedto the original goals. Today, more public healthresearch involves evaluations of interventions toreduce risk in the workplace, and research producesresults that translate into health improvements (R2P).Although it may be difficult to demonstrate reductionin agricultural work-related injuries or illnesses attrib-utable to an intervention, evaluation should be anessential element in measuring past success and deter-mining future priorities.

E. Lee Husting, Ph.DScientific Program Administrator

Agriculture Has Advanced in Technology; Remains High-Risk for Injuries, Illnesses, & Fatalities

2

Evaluating A Community-based ROPS Promotion ProgramThe work reported was completed at the University of Kentucky under CDC/NIOSH Cooperative Agreement U06/CCU412900.1 PI: Hank Cole. The

intervention program materials are available on the NASD website under the Kentucky ROPS Project. This article was abstracted from the 2001 KentuckyROPS Project Final Report by H. P. Cole and S. C. Westneat.

Athree-year community education program was developed andimplemented in two rural Kentucky counties to promotefarmers equipping unguarded tractors with rollover protective

structures (ROPS) and seatbelts. Two other counties served as controls.The program was developed and evaluated in partnership withintervention county community organizations and leaders.

Project outcomes and impact were evaluated through a series ofstudies. These included (1) ethnographic studies of key communityplayers concerning their views of the impact of the program and theirrecommendations for its improvement; (2) monitoring equipment deal-ers’ sales of ROPS in the two intervention counties and then interview-ing the farmers who purchased the ROPS; (3) collecting data fromSAF-T-CAB (a major supplier of retrofit kits) concerning its sales ofROPS to equipment dealers in the intervention and control counties;and (4) repeated measures of analysis of variance (ANOVA) on fourdependent variables obtained from pre- and post-intervention telephonesurvey data gathered from random samples of 26% of 585 interventioncounty and 29.1% of 643 control county farmers. The ANOVAhypotheses were that compared to control county farmers a significant-ly greater proportion of intervention county farmers would:

1. Perceive a higher protective value for ROPS and seatbelts2. Contemplate obtaining ROPS and seat belt protected tractors3. Make efforts to acquire ROPS and seat belts for unguarded tractors4. Retrofit or replace unguarded tractors with

ROPS – equipped tractors The project goals and activities were implemented as planned. The

program materials were widely disseminated, evaluated, and found tostimulate interest in the issue of tractor overturns and the role of ROPSand seatbelts for reducing risk of injury and economic loss. Tractorequipment dealer records revealed a total of four ROPS sales across allfour counties in the year prior to the intervention. Three years and ninemonths from program implementation, dealers in the two interventioncounties had sold 81 ROPS to 79 farmers. For approximately the sameperiod, SAF-T-CAB records showed 50 ROPS sales to equipment deal-ers in the two intervention counties and 10 to dealers in the two controlcounties.

The repeated measures of ANOVA of the telephone survey data con-firmed the first three hypotheses. Intervention county farmers rated the

injury protection value of ROPS higher, exhibited greater contemplationabout acquiring ROPS, and made a greater effort to obtain ROPS than didcontrol farmers. No significant overall difference in the percent of ROPS-equipped tractors acquired was found for intervention and control countyfarms. Intervention and control counties were then individually examinedfor percent of ROPS retrofits acquired (see Table 1). The 95% confidenceintervals revealed that intervention county 02 and control county 04acquired significantly more ROPS tractors than did control county 03.Subsequent investigation revealed that an equipment dealer in controlcounty 04 implemented a successful ROPS-retrofit effort after one of hiscustomers was killed in a tractor overturn. Intervention county 02 was bet-ter organized and more effective in program implementation thanIntervention county 01.

The study illustrates three points. First, community educational interven-tions are likely to be more effective to the degree that community membersare involved in developing and implementing the program. Second, multi-ple evaluation methods are necessary to assess the degree to which a pro-gram is implemented and with what degree of impact. Third, community-trial interventions are never as clean and neat as classical laboratory stud-ies. Replication of both intervention and control groups is desirable. If onlyone control and one intervention county had been used in this study theeffective intervention activities mounted by intervention county 02 and thecontrol county 04 equipment dealer could have obscured the effectivenessof either the planned project intervention and/or the unanticipated equip-ment dealer’s equally effective intervention.

Table 1: Percent increase in ROPS-equipped tractors by county pre- to post-intervention

Intervention ControlCounty ID 01 02 03 04Percent 8.3% 14.8% 5.0% 12.5%95% CI 5.6-12.2 11.0-19.6 3.0-8.1 9.2-16.71 The work reported was completed at the University of Kentucky under

CDC/NIOSH Cooperative Agreement U06/CCU412900. Cole was the PI. Theintervention program materials are available on the NASD website under theKentucky ROPS Project. This article was abstracted from the 2001 KentuckyROPS Project Final Report by H. P. Cole and S. C. Westneat.

3

Questions for Dr. Fleming:

How will the results of the evalua-tion of the NIOSH AgricultureInitiative be used? RF: First is toimprove the delivery of occupa-tional safety and health informa-tion, items, and services to theworker. Dr. Howard’s (NIOSHDirector) Research to Practice (R2P)emphasizes the need to focus ontranslating research findings toactual practices adopted by work-ers. The report will guide plans forenhancements to the agricultureprogram over the next 10 years.

What components of the agricul-ture research program are ofgreatest interest to the reviewpanel? RF: Surveillance,research, intervention, and out-reach are the main areas to beassessed. This isn’t solely aboutthe success of each center individu-ally; the integration of work acrosscenters and a synthesis of thecross-center value of theAgriculture Research Center pro-gram will also be considered.

How and when will the reviewpanel report be distributed? RF:It will be publicly available on theNA website http://www8.nation-alacademies.org/cp/default.aspxin spring of 2007.

How does this evaluation of theNIOSH agriculture initiative com-pare with previous program evalu-ations? RF: This will be morecomprehensive than previous eval-uations. It will cover all intramuraland extramural programs for alonger period of time. In addition,it is focused on intermediate andend outcomes, in addition toprocesses.

How will other programs benefitfrom this evaluation process? RF:The framework designed by theNA review panel establishes amethodology that can be used byprograms in the future to identifythe metrics for evaluating programprogress, value, and success.Researchers and administratorswill be better able to plan theirwork to include intermediate (2-5year) measures and go beyondoutput to actual outcomes that ben-efit workers.

In the fall of 2004 NIOSH funded an evaluation con-tract, submitted by High Plains Intermountain Center

for Agricultural Health and Safety (HICAHS), to under-take both program monitoring and outcome assessmenton ten research, outreach and education Centers collec-tively known as the Agricultural Occupational Healthand Safety Initiative. The Initiative includes 9Agricultural Centers and the National Children’s Centerwhich together undertake research, develop preventionprograms and provide services across ten public healthregions. The mission of these Centers is to reduce injuryand disease in one of the most hazardous occupations inthe United States, agricultural production.

This contract renewed the Agricultural CenterInitiative evaluation effort. Each Center designated a rep-resentative to attend workshops which took place in FortCollins, Colorado during January and June of 2005. TheAgricultural Center Evaluation team (ACE) was chargedwith revising and continuing the program monitoringapproach to document Initiative accomplishments. Inaddition, the contract called for two pilot cross-centeroutcome areas that each includes at least three similarCenter projects. Two project areas have been identifiedby the team as: 1) High School Agricultural Health andSafety Curriculum and 2) Professional development forthose providing health and safety services to agriculturalworkers and their families. A report documenting thefirst fiscal year of this evaluation project was distributedJanuary, 2006.

Agricultural Health and Safety Centers Evaluation ProjectHI-CAHS Center

The staff at HICAHS would like to thank each mem-ber of the ACE team for their time, travel, ideas andefforts in reinstating the Agricultural Health and SafetyCenter Evaluation project. This project is possiblebecause of a truly dedicated and collaborative team.ACE team members include: Sue Ackerman, CynthiaBrundage, Vicky Buchan, Angi Buchanan, RuthieFairbanks, Regina Fisher, Becky Foster, Karen Gilmore,Marcy Harrington, Stacey Holland, Helen Holmquist-Johnson, Lee Husting, Chris Leck, Ketty Mobed, TeriPalermo, Joe Petrik, Wayne Sanderson, Tim Stock, andMarlene Thompson.

How can I receive meaningful feedback from myHispanic participants when it is not possible to inter-

view? This question frames a common dilemma for those of

us outreaching to Hispanic workers. Hispanic farm work-ers are often semi-literate, and with a growing indigenousworkforce, their native language may not be Spanish.

When trying to gain feedback from a large group ofHispanic workers, such as in a training program, we need

Training Program Evaluation with Hispanic WorkersPNASH Center

This evaluation tool asked subjects “What is the most common way handlers are exposed to pesticides while handlingpesticides at work?” The audiotape then gave verbal prompts cued to the answer sheet’s illustrations: mouth, breathing,skin, I don’t know.

Agricultural Researchfrom page 1

to pursue innovative techniques. At the PNASH Centerwe have been using two companion tools with success:

• Illustration-based answer sheets (see below)• Audio tape or CD survey.These self-administered and user-friendly tools meet a

variety of survey needs, but keep in mind that their devel-opment requires iterative audience field-testing. For moreinformation: 206-616-1958, .

Agricultural Research Center Initiative Centers funded as of October 1, 2001

Great Lakes Center for Agricultural Safety and HealthOhio State University590 Woody Hays DriveColumbus, OH 43210(614) 292-9455Center Director: Thomas Bean, EdD

Great Plains Center for Agricultural Health100 Oakdale Campus, #124 IREHThe University of IowaIowa City, IA 52242-5000(319) 335-4887Center Director: Wayne Sanderson, PhD, CIH

High Plains Intermountain Center for Agricultural Health & Safety154 B Environmental Health Bldg.Colorado State UniversityFort Collins, CO 80523-1681(970) 491-6151Center Director: Steve Reynolds, PhD, CIH

National Children’s Center for Rural & Agricultural Health& SafetyNational Farm Medicine Center1000 North Oak AvenueMarshfield, WI 54449-5790800-662-6900Center Director: Barbara C. Lee, PhD

Northeast Center for Agricultural Safety & HealthOne Atwell RoadCooperstown, NY 13326(607) 547-6023Center Director: John May, MD

Pacific Northwest Agricultural Safety & Health CenterDepartment of Environmental Occupational Health SciencesBox 357234University of WashingtonSeattle, WA 98195-1452, (206) 685-8962Center Director: Richard Fenske, PhD, MPH

Southeast Center for Agricultural Health and Injury PreventionDepartment of Preventive MedicineUniversity of Kentucky1141 Red Mile Road, Suite 102Lexington, KY 40504-9842(859) 323-6836Center Director: Robert McKnight, MPH, ScD

Southern Coastal Agromedicine CenterEast Carolina UniversityWest Research Bldg., 1157 VOA Site C RoadGreenville, NC 27858(252) 744-1000Interim Center Director: Kristen Borré, Ph.D.

Southwest Center for Agricultural Health, InjuryPrevention, & EducationThe University of Texas Health Center at Tyler11937 U. S. Hwy. 271Tyler, TX 75708-3154(903) 877-5896Center Director: Jeffrey L. Levin, MD, MSPH

Western Center for Agricultural Health & SafetyUniversity of CaliforniaOne Shields AvenueDavis, CA 95616-8757(530) 752-4050-5253Center Director: Marc Schenker, MD, MPH

NIOSH Program OfficersProgram Administration: E. Lee Husting, PhDSr. Scientist for Agriculture: Steve Olenchock, PhDProgram Specialist: Teri Palermo, RN, MS

Funds for this newsletter provided to the Southwest Center for AgriculturalHealth, Injury Prevention and Education by the National Institute for

Occupational Safety and Health (Contract # 000025170)

4NON PROFITORG

US POSTAGEPAID

TYLER, TXPERMITNO 1028

The University of Texas Health Center at Tyler11937 US Highway 271Tyler, TX 75708-3154ADDRESS SERVICE REQUESTED

NIFS will hold its 2006 annual conference,“Meeting Challenges Together” at the BlueHarbor Resort in Sheboygan, WI June 25-30, 2006. The conference will be hosted bythe National Farm Medicine Center.Several registration options are available,check the internet website for full details,http://www.marshfieldclinic/org./educa-tion/pages/default.aspx

National Library of MedicineUnveils “Tox Town”

Tox Town (http://toxtown.nlm.nih.gov./)announces the release of its newest location, theFarm Scene. Visitors can explore the new Farmneighborhood to learn about the environmentalhealth concerns of agricultural and rural commu-nities. Tox Town illustrates the unique issuesfaced by farms related to hazardous agriculturalchemicals, toxic gases from animal waste, ruraldumping, and farm safety. The Farm features ashed, barn and silo, crop fields, large-scale feed-ing operation, landfill, agricultural runoff, farmpond, meat processing, tree farming and logging,off-road vehicles, and nearby urban sprawl. Eachlocation lists toxic chemicals that might be foundthere and answers questions like “Why are farmponds a concern?” and “What are feeding opera-tions?”

Tox Town is a product of the Division ofSpecialized Information Services (SIS), NationalLibrary of Medicine, National Institutes ofHealth. Professionals with an interest in agricul-tural and environmental concerns are urged totake the time to review this site and provide SISwith your suggestions and comments so that theycan continue to improve the Farm Scene andpresent relevant information.