How To Conduct Surveys - Home | 2017 - UoC Exams | · PDF file · 2017-01-28How To...

HowToConductSurveys6Edition

ThisbookisdedicatedtotheonesIlove:

JohnC.BeckandIngvard

HowToConductSurveysAStep-by-StepGuide

6Edition

ArleneFinkUniversityofCaliforniaatLosAngeles,TheLangleyResearch

Institute

FORINFORMATION:

SAGEPublications,Inc.

2455TellerRoad

ThousandOaks,California91320

E-mail:[email protected]

SAGEPublicationsLtd.

1Oliver’sYard

55CityRoad

LondonEC1Y1SP

UnitedKingdom

SAGEPublicationsIndiaPvt.Ltd.

B1/I1MohanCooperativeIndustrialArea

MathuraRoad,NewDelhi110044

India

SAGEPublicationsAsia-PacificPte.Ltd.

3ChurchStreet

#10-04SamsungHub

Singapore049483

Copyright©2017bySAGEPublications,Inc.

Allrightsreserved.Nopartofthisbookmaybereproducedorutilizedinanyformorbyanymeans,electronicormechanical,includingphotocopying,recording,orbyanyinformationstorageandretrievalsystem,withoutpermissioninwritingfromthepublisher.

PrintedintheUnitedStatesofAmerica

LibraryofCongressCataloging-in-PublicationData

Names:Fink,Arlene,author.

Title:Howtoconductsurveys:astep-by-stepguide/ArleneFink.

Description:Sixthedition.|LosAngeles:SAGE,[2017]|Includesbibliographicalreferencesandindex.

Identifiers:LCCN2015035702|ISBN9781483378480(pbk.:alk.paper)

Subjects:LCSH:Socialsurveys.|Educationalsurveys.

Classification:LCCHN29.F532017|DDC300.72/3—dc23LCrecordavailableathttp://lccn.loc.gov/2015035702

Thisbookisprintedonacid-freepaper.

AcquisitionsEditor:VickiKnight

eLearningEditor:KatieBierach

EditorialAssistant:YvonneMcDuffee

ProductionEditor:JaneHaenel

CopyEditor:KatharineBlankenship

Typesetter:C&MDigitals(P)Ltd.

Proofreader:TriciaCurrie-Knight

Indexer:KathyPaparchontis

CoverDesigner:MichaelDubowe

MarketingManager:NicoleElliott

http://lccn.loc.gov/2015035702

BriefContentsPrefaceAbouttheAuthorChapter1.ConductingSurveys:EveryoneIsDoingItChapter2.TheSurveyForm:Questions,Scales,andAppearanceChapter3.GettingItTogether:SomePracticalConcernsChapter4.SamplingChapter5.SurveyDesign:EnvironmentalControlChapter6.AnalyzingandOrganizingDataFromSurveysChapter7.PresentingtheSurveyResultsIndex

DetailedContentsPrefaceAbouttheAuthorChapter1.ConductingSurveys:EveryoneIsDoingIt

OverviewWhatIsaSurvey?

Example:SurveystoMeetPolicyorProgramNeedsExample:SurveysinEvaluationsofProgramsExample:SurveysforResearch

WhenIsaSurveyBest?Example:FromanOverlyAmbitiousSelf-AdministeredQuestionnaireExample:SurveysCombinedWithOtherInformationSources

Self-AdministeredQuestionnairesandInterviews:TheHeartoftheMatterQuestionsandResponsesExample:Forced-ChoiceQuestionExample:Open-EndedQuestionSurveySampleandDesignPlanningforDataAnalysisPilotTestingResponseRateReportingResultsExample:TheLookofSurveyResults(1)Example:TheLookofSurveyResults(2)Example:TheLookofSurveyResults(3)

TheFriendlyCompetitionReliabilityandValidityUsefulnessorCredibilityofResultsCostsTheSpecialCaseofOnlineSurveysChecklistforDecidingtheBestUsesforOnlineSurveysGuidelinesforSurveyorsWhoWorkWithCommercialSurveyCompaniesTheSpecialCaseofCellorMobilePhonesMakingtheDecision

ASurveyContinuum:FromSpecifictoGeneralUseExample:SurveyWithaSpecificUseExample:SurveyWithaGeneralUse

Ethics,Privacy,andConfidentialityInformedConsentContentsofanInformed-ConsentFormTheInternetandEthicalSurveysExampleQuestionnaire:MaintaininganEthicallySoundOnlineSurveyExample:Informed-ConsentFormforanOnlineSurvey

ChildrenandSurveyEthicsExample:ChildAssentForm

InternationalSurveysFormalStandardsforSurveyEthicsSummingUpThinkAboutThisReferences

Chapter2.TheSurveyForm:Questions,Scales,andAppearanceOverviewTheContentIstheMessageDefinetheTermsSelectYourInformationNeedsorHypothesesMakeSureYouCanGettheInformationDoNotAskforInformationUnlessYouCanActonIt

Example:PlanforSurveyofSatisfactionWiththeYouthCenterWritingQuestions

Open-EndedandClosedQuestionsExample:Open-EndedQuestionExample:ClosedQuestionExample:Open-EndedQuestionforElementarySchoolTeachingProgramExample:ClosedQuestionforElementarySchoolTeachingProgramMakingtheDecision:Open-EndedVersusClosedQuestions

OrganizingResponsestoOpen-EndedSurveyItems:DoYouGetAnySatisfaction?Step1:AskingRespondents’OpinionsStep2:CodingLB/LLDataExampleLB/LL:ResponseCategoriesExampleLB/LL:ParticipantResponsesStep3:LB/LLDataExampleLB/LL:NumberofResponsesforEachCodeExampleLB/LL:Participants’ResponsePatternExampleLB/LL:SummaryofResponses

RulesforWritingClosedSurveyQuestionsExample:Item-WritingSkills—Length,Clarity,Abbreviations,andJargonExample:Item-WritingSkills—ConcreteQuestionsExample:Item-WritingSkills—SpecificityofQuestionsExample:Item-WritingSkills—HiddenBiasesExample:Item-WritingSkills—HiddenBiasesExample:Question-WritingSkills—VeryPersonalQuestionsExample:Question-WritingSkills—OneThoughtperQuestion

ResponsesforClosedQuestionsYesandNoExample:Yes-and-NoResponses

RatingScalesCategoricalorContinuous?WhatAboutOrdinal?Example:CategoricalRatingScaleExample:OrdinalRatingScaleOrdinalScalesExample:OrdinalScalesExample:SelectingtheNumberofCategories

Example:RankOrderScaleExample:ComparativeRatingScaleChecklistExample:ChecklistResponsesinWhichRespondentMustChooseOneFromaListofSeveralExample:ChecklistResponsesThatRespondentsAnswerYes,No,orDon’tKnowforEachIteminaList

ChildrenandSurveysOnlineSurveys

Example:SurveyProgressBarExample:ResponsetoOneQuestionDictatestheResponsetotheNextExample:ExplanationforChangingOnlineResponses

PlainandSimpleSurveyQuestionsandResponsesScaling

AdditiveScalesExample:ASurveyWithanAdditiveScaleExample:ASurveyofForeignLanguageSkillsDifferentialScalesExample:ScoringaDifferentialScaleSummatedScalesExample:CreatingaSummatedScaleforaSelf-EsteemSurveyExample:ScoringaSummatedScale

SummingUpThinkAboutThis

Chapter3.GettingItTogether:SomePracticalConcernsOverviewLengthCounts

Example:HowaSurvey’sCircumstancesCanInfluenceItsLengthGettingtheSurveyinOrder

Example:AnIntroductiontoaTelephoneSurveyandItsFirstQuestionExample:OrderingSurveyQuestions(1)Example:OrderingSurveyQuestions(2)Example:ProvidingTransitionsChecklisttoGuideQuestionOrder

QuestionnaireFormat:AestheticsandOtherConcernsResponseFormatExample:ResponseFormats

BranchingQuestions,ortheInfamous“Skip”PatternExample:SkipPatternsorBranchingQuestions

Administration:WhoGivesWhattoWhom?Self-AdministeredQuestionnairesChecklistforUsingSelf-AdministeredQuestionnairesInterviews

ReliabilityandValidity:TheQualityofYourSurveyIstheSurveyReliable?Example:InternalConsistencyCountsExample:InternalConsistencyDoesNotCount

IstheSurveyValid?SelectingandAdaptingSurveys

Example:ExcerptFromaRealOnlineServiceAgreementforFreeSurveyItems(NamesChanged)GuidelinesforFindingUseableandUsefulSurveysintheResearchLiteratureExample:SearchforSchoolDropoutSurveys:SampleFindingsFromtheWebofScienceExample:SampleReportsofReliabilityandValidityintheResearchLiterature

FindingSurveysontheWebExample:SearchforEducationSurveys:ASamplePageFromtheNationalCenterforEducationStatistics

TheSurveyIsPutonTrial:GuidelinesforPilotTestingAFar-ReachingWorld:Surveys,Language,andCulture

GuidelinesforTranslatingSurveysExample:QuestionAboutEthnicity:U.S.DepartmentofHealthandHumanServicesOfficeofMinorityHealth

SummingUpThinkAboutThisReferences

Chapter4.SamplingOverviewSampleSizeandResponseRate:WhoandHowMany?

Example:RandomandConvenienceSamplingRandomSamplingMethods

Example:SimpleRandomSampling(1)Example:NotRandomSamplingExample:SimpleRandomSampling(2)MakingtheDecision

StratifiedRandomSamplingExample:StratifiedRandomSamplingMakingtheDecision

SimpleRandomClusterSamplingExample:SimpleRandomClusterSamplingMakingtheDecision

SystematicSamplingMakingtheDecision

ConvenienceSamplesMakingtheDecision

OtherConvenienceSamplingMethodsExample:OtherConvenienceSamples

FindingtheSample:WhoIsIn?WhoIsOut?HowLargeShouldYourSampleBe?

TheStandardErrorStatisticalMethods:SamplingforTwoGroupsandanIntervention

Example:SampleSizeCalculationsforSamplingTwoGroupsandanInterventionSubgroups,Measures,Resources,andScheduleFiveQuestionstoAskWhenDeterminingSampleSizeExample:CalculatingSampleSizeinaSurveyofEmployeesinanExperimentaland

ControlGroupExample:PowertoDetectDifferencesExample:80%PowerandEffectExample:SampleSize,Effect,andPower

ResponseRateTipsforImprovingResponseRate

WeightingExample:CalculatingPoststratificationWeights

MarginofErrorandConfidenceLevelSampleSizeandtheMarginofError


Chapter5.SurveyDesign:EnvironmentalControlOverviewWhichDesignsAreAvailable?

Example:SurveysWithDifferingDesignsCross-SectionalSurveyDesigns

Example:Cross-SectionalDesignMakingtheDecision

LongitudinalSurveysExample:LongitudinalSurveys:CohortDesignMakingtheDecision

ExperimentalSurveyDesignsExample:ANonrandomizedControlledTrialorQuasi-ExperimentalDesignExample:ANonrandomizedControlledTrialWithaLongitudinalDesignExample:ARandomizedControlledTrialorTrueExperimentExample:ARandomizedControlledTrialWithaLongitudinalDesignFactorialDesigns:SpecialCasesMakingtheDecision

OtherSurveyDesigns:NormativeandCaseControlNormativeSurveyDesignExample:NormativeDesignExample:NormativeDesign—ComparisontoaModelMakingtheDecision

CaseControlDesignMakingtheDecisionExample:CaseControlDesign

SurveyDesignValidityInternalValidityExternalValidity

Surveys,ResearchDesign,andInternalandExternalValidityExample:Cross-SectionalSurveyandThreatstoValidityExample:LongitudinalDesignandThreatstoValidityExample:RandomizedControlledTrialsandValidity

SurveysWithQualitativeData:ThreatstoInternalandExternalValidityExample:QualitativeStudiesandValidity


Chapter6.AnalyzingandOrganizingDataFromSurveysOverviewWhatIsTypicalAnyway?SomeCommonlyUsedMethodsforAnalyzingSurveyData

DescriptiveStatisticsExample:PreschoolPurposesQuestionnaireExample:FrequencyofQuestionnaireResponsesExample:GroupedRatingsofPreschoolPurposesby50DirectorsAverages:Means,Medians,andModesExample:ComputingtheMedianforanEvenNumberofScoresExample:ComputingtheMedianforanOddNumberofScoresVariation:Range,Variance,andStandardDeviationCorrelationandRegressionExample:RankOrderCorrelationDifferencesBetweenGroupsStatisticalDifferencesStatisticalSignificance

SurveyingDifferences:UsualMethodsChi-SquareExample:Chi-SquareThetTestExample:tTestTheMann-WhitneyUTestExample:Mann-WhitneyUTestANOVARisksandOddsExample:OddsRatioandRelativeRisk

ToBeorNottoBe:StatisticianorQualitativeAnalyst?ContentAnalysis,Open-EndedResponses,andComments

Example:HypotheticalContentAnalysis:TeasingBoysandGirlsPuttingtheCartinFrontoftheHorse:SelectingAnalysisMethods

SevenQuestionstoAnswerBeforeChoosinganAnalysisMethodDataManagement

CreatingaCodeBookorOperationsManualExample:ExcerptFromtheCARPS,aSurveytoDetectBingeDrinkinginCollegeStudentsExample:ExcerptFromtheCodeBookfortheCARPS,aSurveytoDetectBingeDrinkingEstablishingReliableCodingMeasuringAgreement:TheKappaMeasuringAgreementBetweenTwoCoders:TheKappa(κ)StatisticReviewingSurveysforMissingDataEnteringtheDataExample:SurveyResponsesFromSixPeopleCleaningtheDataValidatingSurveyData

SummingUpThinkAboutThis

Chapter7.PresentingtheSurveyResultsOverviewReproducingtheQuestionnaire

Example:ReportingResultsWiththeQuestionnaireExample:OnlineSurveyStatisticsinRealTime

UsingTablesExample:ShellTableDescribingChildreninTwoSchoolsExample:ShellTableforComparingChildreninTwoSchoolsSomeTablePreparationRules

DrawingPieDiagramsUsingBarGraphsUsingLineGraphs

Example:DownloadedResultsofaCustomerSatisfactionSurveyExample:TransferringtheResultsofOneQuestionIntoBarandLineGraphs

DrawingDiagramsorPicturesExample:WordsandDiagramsinSurveyReports

WritingtheResultsofaSurveyOrganizingtheReportExample:StructuredAbstractofaSurveyReport

SurveyReportingChecklistsandGuidesClear-WritingTips

TheOralPresentationSlidePresentationsOralVersusWrittenReports:ADifferenceinConversation

Example:TableUsedinaWrittenandanOralReportPosters


Index

Preface

Thesixtheditionofthisbooksharesthesamegoalsasthefirst,second,third,fourth,andfiftheditions:toguidereadersindevelopingtheirownrigoroussurveysandinevaluatingthecredibilityandusefulnessofsurveyscreatedbyothers.Thisedition,likepreviouseditions,alsogivespracticalstep-by-stepguidanceonhowtoachievethegoals.Theguidancecentersonchoosingtheappropriatetypeofsurvey,writingsurveyquestionsandresponses,formattingthesurvey,decidingonthecharacteristicsandnumbersofrespondentstoincludeandhowoftentheyshouldbesurveyed,andanalyzingandreportingtheresults.

Thiseditionalsosharesthesamephilosophyaspreviouseditions.Itemphasizestheneedforcarefulplanningandtestingbeforeyoupress“send,”mailthefirstsurvey,orconductthefirstinterview.Butthesimilaritiesamongtheeditionsendhere,andeachchapterinthesixtheditionhassomethingthatyouwon’tfindintheothereditions.

Thefiftheditionplacedastrongemphasisonweb-basedsurveys.Atthattime,web-basedsurveyswerebeginningtheirclimbtothetop.Ithinkwecansaywithoutcontradictionthatinternetsurveyshavemadeittothetop,andsothiseditionaddsmoreabouthowtodothesesurveysandhowtohandleprivacy.Theuniversalityofwebsurveyshasraisedsomenewconcernsamongresearchers,amongthem,howtohandleinternationalsurveys.Afterall,withtheweb,youcanreachuntoldnumbersofpeopleandtheycanbelivinganywhereontheplanet.Butculturalbeliefsandexpectationsvarywidely,andthesurveyorcannotassumethatquestionsthat“work”inoneculturewillnecessarilybeeffectiveinanother.Inthiseditionofthebook,weaddressthepracticalandethicalissuesofgoingbeyondourbordersforsurveyinformation.

Also,sinceyoungpeoplearesocomfortablewithsurveysofallsorts,thisneweditionofthebookprovidesinformationonhowtoquestionchildrenofallages,anditaddressestheethicalconcernsthatarisewhenworkingwithyoungpeople.

Thesixtheditionalsorecognizesanewtrendinreportingresearchresults.Surveyorsarenowrequiredtoadheretoveryspecificguidelinesconcerningthecontentsofareport.Peoplewhoreadandusesurveyresultswanttoknowmorethanjustwhatyoufound.Theywanttobesurethattheinformationcomesfromreliablesources,andtheywanttoknowhowconfidenttheycanbeinyourresults.Theywanttransparency.Thesixtheditionofthisbookgivesyoureportingcheckliststofollow,anditdiscusseshowtodescribehowmuchconfidencesurveyuserscanplaceinyoursurveyresults.

Herearesomeofthenewtopicsweexploreinthesixthedition.

ChildrenandSurveys.

Childrenofallagesarenowincludedinsurveys.Wewanttoknowhowchildrenspendtheirsparetime,whethertheyarehappyathomeandschool,andwhethertheydrink,smoke,oruseillicitdrugs.Childrenvaryintheirageandabilitytodealwithabstractquestions.Theneweditionofthisbookanswersquestionslike:Howdoesasurveyreachchildrentogetreliableandvalidinformation?Howdoesasurveyforyoungchildrendifferfromdoingoneforteens?

ChildrenandSurveyEthics.

Ethicsrequirementsforchildrenareoftenstricterthanthoseforadults.Thiseditionofthebookaddresseshowtoaccommodateethicsboardsanddiscussesthedifferencebetweenassentandconsentinchildren.

QualitativeSurveyData:ValidityandReporting.

Mostsurveysproducestatisticaldata,butothersmayincludeopen-endedquestionsthatneedinterpretation.Theymayrelyonsemistructuredorevenunstructuredinterviews,forexample.Manyproblemsarebettersolvedwithacombinationofsurveydesignsandmethods.Thiseditiontothebookdiscussesthevaluesofand“threats”tothevalidityofqualitativestudiesandprovidesguidanceintransparentlyreportingtheirpurposesandmethods.

InternationalSurveys.

WiththecomingofsmartphonesandWi-Fi,surveysareincreasinglybecominginternational.Inthiseditionofthebook,weaddresshowtoputintopracticeourrespectforculturaldifferencesindesigningthesurveyandinaskingquestions.

MarginofErrorandConfidenceLevels.

Allsurveyshaveerrorsbecauseofimperfectsampling.Thelargerthemarginoferror,thelessconfidencewecanhavethatthesurvey’sresultsareclosetothe“true”figures.Thiseditiondiscusseshowtothinkaboutandcalculatethemarginoferrorandconfidencelevelforyoursurvey.

SurveyReportChecklists.

Onceuponatime,surveyorscouldissuetheirreportsinanywaytheysawfit.Now,thatisnolongertrue,especiallyforpublishedsurveys.Manyjournals,businesses,andeducationalinstitutionsrequiresurveyorstoprovethattheyhavecompletedstandardchecklists.Adheringtothesechecklistsisthoughttobetterensurethetransparencyandaccuracyoftheirmethods.Thiseditionofthebookdiscussesthesechecklistsandtheircontents.

Thesixtheditionofthisbookalsohashadafacelift.Wehaveenlargedthefontstomakeforeasierreadingandretooledthegraphics.Ihaveleftsomefeaturesofthesixtheditionexactlyastheyareintheothereditionsbecausereadersandreviewersfindthemuseful.Theseincludenumerousexamplesandchecklistsandthe“SummingUp”sectionateachchapter’sendtomakesurethatthereaderhasaccesstothemostimportantconcepts.Ihavekeptthesummariesbrief,butIthinktheyareworthwhileandthatitishelpfulforpeopletoknowthebottomline:WhatshouldIfocuson?Toreinforceallmajorpoints,eachchapterstillhaspracticeexercises,manyofthemnewtothesixthedition.Theexerciseswillleadyoutosurveyreportsandtoresearchthatusessurveyssothatyoucantestyourskillsinunderstandingandevaluatingthem.Mostimportantly,Ihavegearedthepracticeexercisestoengageandpromotethereader’sskillsofanalyzing,evaluating,andproducinginformation.Thismeansavoidingexercisesthatfocusonjustregurgitatingfacts.Ihavealsoexpandedandupdatedthechapterbibliographiesandaddedadditionalwebsites,surveyreports,andresearchstudiescontainingsurveys.Ihavelistedsitesforyoutoexploreincaseyouwanttoadaptquestionsfromothersurveys.Finally,thesurveyormustmakechoicesthroughouttheentiresurveyprocess.Inresponsetothissometimesdifficultsurveyactivity,Ihavekeptthe

sectionsfoundthroughoutthebookcalled“MakingtheDecision.”Thepurposeofthesesectionsistohelpthereadertomakeinformedchoicesbycitingtheadvantagesanddisadvantagesateachwaypoint.Youcancontinuetoexpectguidanceinchoosingsurveytypes,questiontypes,samplingstrategies,researchdesigns,anddata-analyticmethods.

Ihaveorientedthisbooksothatitcanbeusedbyeveryonewhoneedstolearnhowtodoasurvey,regardlessofbackground.Thebookislongonpracticeandshortontheory.Itisintendedtohelpyoudoasurvey.Ifyouwantsurveytheory,pleaseusethebook’sbibliography.Ithinkthebookisusefulforself-learningandinworkshops,seminars,andformalclassrooms.

Onapersonalnote,Iwouldliketothankmyeditor,VickiKnight,forsupportingmethroughouttherevisionprocess.ShesawtoitthatweassembledalargepanelofreviewerstowhomIamdeeplyindebted:

ColleenL.Casey,UniversityofTexasatArlingtonSalvatoreFalletta,DrexelUniversityRheaFayeD.Felicilda,MissouriStateUniversityMichaelA.Guerra,LincolnUniversityHowardLune,HunterCollege,CUNYB.J.Moore,CaliforniaStateUniversity,BakersfieldDawnM.Pickard,OaklandUniversityMarcS.Schwerdt,LipscombUniversityJulieSlayton,UniversityofSouthernCalifornia

Theirsuggestionswereoutstanding,andbecauseIamtrulygratefulfortheirtimeandeffort,Ireallydidtrytoincorporatetheirsuggestions.WhenIdidn’t,itwasusuallyjustamatterofconvenience(mine),andIamsurethebooksuffersbecauseofit.IwouldalsoliketothanktheSAGEProductionteamfortheirhelpinreformattingandrecastingthebook.Itismuchbetter,andtheyareresponsiblefortheimprovement.

AbouttheAuthor

ArleneFink(PhD)isProfessorofMedicineandPublicHealthattheUniversityofCalifornia,LosAngeles,andpresidentoftheLangleyResearchInstitute.Hermaininterestsincludeevaluationandsurveyresearchandtheconductofresearchliteraturereviewsaswellastheevaluationoftheirquality.Dr.Finkhasconductedscoresofevaluationstudiesinpublichealth,medicine,andeducation.SheisonthefacultyofUCLA’sRobertWoodJohnsonClinicalScholarsProgramandisascientificandevaluationadvisortoUCLA’sGamblingStudiesandIMPACT(ImprovingAccess,Counseling&TreatmentforCalifornianswithProstateCancer)programs.SheconsultsnationallyandinternationallyforagenciessuchasL’institutdePromotiondellaPréventionSecondaireenAddictologie(IPPSA)inParis,France,andPeninsulaHealthinVictoria,Australia.ProfessorFinkhastaughtandlecturedextensivelyallovertheworldandistheauthorofmorethan130peer-reviewedarticlesand15textbooks.

1CONDUCTINGSURVEYSEveryoneIsDoingIt

OverviewSurveysareeverywhere.Youwillfindthemindoctors’offices,schools,airplanes,andhotelrooms.Surveysareusedtocollectinformationfromoraboutpeopletodescribe,compare,orexplaintheirknowledge,feelings,values,andbehavior.Surveystypicallytaketheformofself-administeredquestionnairesandinterviews.Self-administeredquestionnairescanbecompletedbyhand(“paperandpencil”)oronline(Internetorweb-basedoncomputersandtablets).Interviewsmaytakeplaceinperson(“face-to-face”),onthetelephone(landlineandmobile/cell),andinchatrooms.

Surveydataareusedbyprogramplanners,evaluators,administrators,managers,researchers,marketers,andpolicyleadersindiversefields,includingbusiness,health,education,socialwelfare,andpolitics.Theyareusedbecausetheygetinformationdirectlyfrompeople.

Surveyorsmustdecideonthesurvey’soverallpurposesandspecificquestions.Theyalsoneedtoknowwhoandhowmanypeoplewillbecontacted(sampling)andwhenandhowoftenthesurveywilltakeplace(design).Surveyorsmustalsoprocess,analyze,andinterpretdata.

Choosingamongsurveytypes(self-administeredquestionnairesorinterviews)andadministrationmethods(mail,telephone,orinternet)requires(1)identifyingthecombinationmostlikelytoproducecredibleandaccurateresultsand(2)balancingthedesiredsurveytypesandadministrationmethodsagainstavailableresources.

Surveypurposesandmethodsfallonacontinuum.Somesurveyscanhavefar-reaching,generalizableeffects,andtheirmethodsmustbescientific.Surveysofthepopulation’shealthconductedbytheU.S.governmentareexamplesofscientificsurveys.Othersurveysareconductedtomeetspecificneeds;theirmethodsmaynotalwaysachievethehigheststandardsofscientificrigor,buttheymuststillproduceaccurateresultsandsomustusereliableandvalidtechniques.Pollingstudentsinaparticularschooltoidentifytheirsummerreadingchoices,soastobesurethelibraryiswellstocked,isanillustrationofasurveydesignedtomeetaspecificneed.

Surveyorsmustbeconcernedwithprotectingrespondents’privacyandassuringconfidentialityofresponses.Mostinstitutionshaverulesforprotecting“humansubjects”andensuringthattheirconsenttorespondisaninformedconsent.Onlinesurveyshavespecialrulesforguardingconfidentiality.Thesesurveysmaybevulnerabletooutsidersandneedprotectionacrossthreecommunicationarms:surveyortorespondent,respondenttowebserver,andwebservertosurveyor.

WhatIsaSurvey?Surveysareinformationcollectionmethodsusedtodescribe,compare,orexplainindividualandsocietalknowledge,feelings,values,preferences,andbehavior.Asurveycanbeaself-administeredquestionnairethatsomeonefillsoutaloneorwithassistance,orasurveycanbeaninterviewdoneinpersonoronthetelephone.Somesurveysareonpaperoronline,andtherespondentcancompletethemprivatelyathomeorinacentrallocation—say,atahealthcenter.Therespondentcaneitherreturnthecompletedsurveybysnailmailorfillitoutonline.Surveyscanbeinteractiveandguidetherespondentthroughthequestions.Interactivesurveysalsomayprovideaudiovisualcuestohelp.

Hereareatleastthreegoodreasonsforconductingsurveys:

Reason1:Apolicyneedstobesetoraprogrammustbeplanned.

Example:SurveystoMeetPolicyorProgramNeeds

TheYMCCorporationwantstodeterminewhichhourstobeopeneachday.Thecorporationsurveysemployeestofindoutwhich8-hourshiftstheyarewillingtowork.ThenationalofficeoftheHealthVoluntaryAgencyisconsideringprovidingdaycareforitsstaff’schildren.Howmanyhaveveryyoungchildren?Howmanywouldusetheagency’sfacility?Tenyearsago,theBartleySchoolDistrictchangeditslanguageartscurriculum.Sincethen,somepeoplehavearguedthatthecurriculumhasbecomeoutofdate.WhatdotheEnglishteachersthink?Ifrevisionsareneeded,whatshouldtheylooklike?

Reason2:Youwanttoevaluatetheeffectivenessofprogramstochangepeople’sknowledge,attitudes,health,orwelfare.

Example:SurveysinEvaluationsofProgramsTheYMCCorporationhascreatedtwoprogramstoeducatepeopleabouttheadvantagesanddisadvantagesofworkingatunusualhours.Oneprogramtakestheformofindividualcounselingandspeciallyprepared,self-monitoredDVDs.Thesecondprogramisconductedinlargegroups.Asurveyisconducted6monthsaftereachprogramiscompletedtofindoutwhethertheemployeesthinktheygottheinformationtheyneeded.Thesurveyalsoaimstofindoutwhethertheywouldrecommendthatothersparticipateinasimilarprogramandhowsatisfiedtheyarewiththeirworkschedule.TheHealthVoluntaryAgencyistryingtwoapproachestochildcare.Oneisprimarily“childcentered,”andthechildrenusuallydecidefromalistofactivitieswhichonestheywouldliketododuringthehourstheyareintheprogram.Theotherisacademicandartistic.Childrenaretaughttoread,playmusicalinstruments,anddanceatsettimesduringtheday.Whichprogramismostsatisfactoryinthattheparents,children,andstaffareactiveparticipantsandpleasedwiththecurriculum’scontent?Theagencysurveysparents,children,andstafftogetanswers.TheBartleySchoolDistrictchangeditslanguageartscurriculum.Asurveyisconductedtofindoutwhetherandhowthechangehasaffectedparents’andstudents’opinionsofthehighschoolprogram.

Reason3:Youarearesearcherwhousesasurveytogetinformationabouthowtoguidestudiesandprograms.

Example:SurveysforResearchBecausetheYMCCorporationhassomanyeducationalprograms,itwantstoresearchhowadultslearnbest.Dotheypreferself-learningorformalclasses?Arereadingmaterialsappropriate,orarefilmsandDVDsbetter?Howdotheyfeelaboutcomputer-assistedlearningorlearningdirectlyfromtheInternet?Aspartofitsresearch,andtomakesureallthepossibilitiesarecovered,thecorporationconductsasurveyofasampleofemployeestolearntheirpreferences.TheHealthVoluntaryAgencyisconsideringjoiningwithalocaluniversityinastudyofpreschooleducation.Theagencyconductsasurveyoftheparentsparticipatinginthenewdaycareprograms.Thesurveyasksabouttheparticipants’educationandincome.Researchersneeddatasuchasthesesotheycantestoneoftheirmajorassumptions—namely,thatparentswithhighereducationandincomesaremorelikelytochoosethemoreacademicofthetwopreschoolprograms.TheBartleySchoolDistrictispartofaU.S.government–fundednationalstudyoftheteachingoftheEnglishlanguage.Thestudy’sresearchershypothesizedthatclassroomteachingdependsmoreontheirteachers’educationalbackgroundsandreadingpreferencesthanontheformalcurriculum.Asurveyisconductedtofindoutteachers’educationalbackgroundsandreadinghabitssothatthosedataareavailablefortestingtheresearchers’hypothesis.

WhenIsaSurveyBest?Manymethodsexistforobtaininginformationaboutpeople.Asurveyisonlyone.Considertheyouthcenterthathasasitsmajoraimtoprovideavarietyofservicestothecommunity.Itoffersmedical,financial,legal,andeducationalassistancetoresidentsofthecitywhoarebetween12and21yearsofage,regardlessofbackground.Theprogramisparticularlyproudofitscoordinatedapproach,arguingthatthecenter’seffectivenesscomesfrommakingavailablemanyservicesinonelocationtoall

participants.Nowthatthecenteris10yearsold,asurveyistobeconductedtofindoutjusthowsuccessfulitreallyis.Areparticipantsandstaffsatisfied?Whichservicesdoyoungpeopleuse?Isthecenterreallyamultiserviceone?Arepeoplebetteroffwiththeirhealthandotherneedsbecauseoftheirparticipationinthecenter?Amailedself-administeredquestionnairesurveyisdecidedontohelpanswertheseandotherquestions.Herearesomeexcerptsfromthequestionnaire.

ThequestionnairewasshowntoareviewerwhoseadvicewastoeliminateQuestions5,7,and11,andkeeponlyQuestion10.Thereviewerstatedthatsurveysarenotbestforcertaintypesofinformation.Hereisthereasoning:

Question5asksforareportofaperson’sbloodpressure.Isitnormal?Ingeneral,informationofthiskindismostaccurateifitisobtainedfromothersources—say,amedicalrecord.Manypeoplemighthavedifficultyrecallingtheirbloodpressurewithprecisionandalsowouldbeatalosstodefinenormalbloodpressure.

Question7maybeallrightifyoufeelconfidentthattheperson’srecallwillbeaccurate.Otherwise,therecordsofthecenterareprobablyabettersourceofinformationaboutwhichservicesareused.

Question11asksthepatienttotellhowmuchtimethedoctorspendslisteningratherthantalking.Ifyouareinterestedinthepatient’sperceptions,thenthequestionisfine.If,however,youwantdataontheactualtimethedoctorlistenedratherthantalkedtothepatient,observationbyanimpartialobserverisprobablybest.

Question10isappropriate.Onlyparticipantscantellyouhowsatisfiedtheyare.Noothersourcewilldosoaswell.

Surveysarebynomeanstheonlysourceofinformationformakingdecisions,noraretheynecessarilythemostrelevant.Someothersourcesofinformationarethefollowing:

Observationsoreyewitnessreports;filmed,videotaped,andaudiotapedaccountsPerformanceteststhatrequireapersontoperformatask(suchasteachingalessontoaclass);observersassesstheeffectivenessoftheperformanceWrittentestsofabilityorknowledgeReviewsthatrelyonexistingdocumentation,suchasexaminationofmedicalandschoolattendancerecords;analysisofthecontentofpublishedandunpublishedarticlesanddiaries;interpretationoftheactivitiesofonlinechatandsupportgroups

Surveyscanbeusedindecidingpolicyorinplanningandevaluatingprogramsandconductingresearchwhentheinformationyouneedshouldcomedirectlyfrompeople.Thedatatheyprovidearedescriptionsoffeelingsandperceptions,values,habits,andpersonalbackgroundordemographiccharacteristicssuchasage,health,education,andincome.

Sometimessurveysarecombinedwithothersourcesofinformation.Thisisparticularlytrueforevaluationsandresearch.

Example:FromanOverlyAmbitiousSelf-Administered

Questionnaire5.Isyourbloodpressurenownormal?

Yes 1No 2

7.Whichofthefollowingsocialserviceshaveyouusedinthelast12months?(Pleaseindicateyesornoforeachservice.)

10.Howsatisfiedareyouwitheachofthefollowingservices?Pleaseindicateyoursatisfactionforeachservice.

11.Howmuchtimeina5-minuteperioddoesthedoctorspendlistening(ratherthan,say,talking)toyou?(Pleasecheckone.)Lessthan1minuteAbout1or2minutesMorethan2minutes

Example:SurveysCombinedWithOtherInformationSourcesAspartofitsevaluationofchildcareprograms,theHealthVoluntaryAgencysurveyedparents,children,andstaffabouttheirdegreeofparticipationandsatisfaction.Also,theagencyreviewedfinancialrecordstoevaluatethecostsofeachprogramandstandardizedtestsweregiventoappraisehowreadychildrenwereforschool.TheYMCCorporationisresearchinghowadultslearn.Achievementandperformancetestsaregivenatregularintervals.Inaddition,asurveyprovidessupplementaldataonhowadultsliketolearn.

Self-AdministeredQuestionnairesandInterviews:TheHeartoftheMatter

Allsurveysconsistof(1)questionsandresponses.Togetaccuratedata,youmustaccountforasurvey’s(2)samplinganddesign,(3)dataprocessingor“management”andanalysis,(4)pilottesting,and(5)responserate.Surveyresultsarepresentedaswritten(printedonpaperorreproducedelectronically)andoralreports.

QuestionsandResponsesInformationfromsurveysisobtainedbyaskingquestions.Questionsaresometimesreferredtoasitems.Thequestionsmayhaveforced-responsechoices.

Theselection,wording,andorderingofquestionsandanswersrequirecarefulthoughtandareasonablecommandoflanguage.

Example:Forced-ChoiceQuestionWhatisthemainadvantageofmultiple-choiceoveressayquestions?

CanbescoredobjectivelyArebestatmeasuringcomplexbehaviorsCanhavemorethanoneanswerAretheleastthreateningofthequestiontypes

Questionsonsurveysmaybeopenended.

Example:Open-EndedQuestionWhatisthemainadvantageofmultiple-choiceoveressayquestions?

Answerhere:

____________________________________________________________________________

____________________________________________________________________________

____________________________________________________________________________

____________________________________________________________________________

____________________________________________________________________________

SurveySampleandDesignSurveysaredatacollectionmethodsusedtoobtaininformationfromandaboutpeople:Fromandaboutwhichpeople,howoften,andwhen?Assoonasyouraisequestionssuchasthese,youmustbecomeconcernedwiththesampleanddesignofthesurvey.Thesampleisthenumberandcharacteristicsofpeopleinthesurvey.Thedesignreferstohowoftenthesurveytakesplace(justonce,orcross-sectional;overtime,orlongitudinal),whethertheparticipantsareselectedatrandomorarechosensomeotherway,andhowmanyseparategroupsareincluded.

Considerthesethreesurveys:

Survey1:Whatdograduatesfromtheclassof2015knowaboutphysicalfitness?

Surveymethod:Onlinequestion-naireSample:All1,000graduatesfromStateCollege’sclassof2015Howoftensurveytakesplace:Justonce—atgraduationHowparticipantsareselected:AllgraduatesareeligibleHowmanygroups:Justone—theclassof2015Design:Cross-sectional

Survey2:Doesknowledgeaboutphysicalfitnesschangeovera12-monthperiodamonggraduatesoftheclassof2015?

Surveymethod:Onlinequestion-naireSample:All1,000graduatesfromStateCollege’sclassof2015Howoftensurveytakesplace:Twice—atgraduationand12monthslaterHowparticipantsareselected:AllgraduatesareeligibleHowmanygroups:Justone—theclassof2015Design:Longitudinalcohort

Survey3:Overtime,dodifferencesexistamonggraduatingclassesintheirknowledgeofphysicalfitness?

Surveymethod:Onlinequestion-naireSample:A75%randomlyselectedsampleofgraduatesfromtheclassesof2015,2017,and2019toequal2,250graduatesHowoftensurveytakesplace:Threetimes—atgraduationand12and24monthslaterHowparticipantsareselected:RandomlyHowmanygroups:Three—theclassesof2015,2017,and2019Design:Longitudinalandcomparative

Survey1asksforaportraitoftheclassof2015’sknowledgeofphysicalfitness,andamailedquestionnaireistobeused.Thisportraitiscalledacross-sectionalsurveydesign.Survey2wantstoknowaboutchangesinknowledgeofphysicalfitnessovera1-yearperiod:fromgraduationforward12months.Thedesignislongitudinal.Theentireclassisa“cohort”ofpeople.

Survey3islongitudinalbecausesurveydataarecollectedfromeachofthethreegraduatingclassesoverthreepointsintime:atthetimeofgraduationand1and2yearslater.Thedesignalsoiscomparativebecauseknowledgecanbecomparedbetweenanytwoandamongallthreeclassesatgraduation,1yearlater,2yearslater,oracrossallthreetimes.AnillustrationofthedesignforSurvey3cantaketheformofTable1.1.

Survey3differsfromSurveys1and2inhowthegraduatesareselectedforparticipation.InSurvey3,a75%sampleofgraduateswillberandomlyselectedtoparticipate.Intheothertwosurveys,allgraduates,notjustasample,areeligible.Randomselectionmeansthateachgraduatehasanequalchanceofbeingincluded.

Allthreesurveysareonline,buttheirsamplesanddesignsvary.

PlanningforDataAnalysisRegardlessofyoursurvey’sdesignorsize,youmustthinkaheadtohowyouplantoanalyzethesurvey’sdata.

Willyoucomputepercentagessothatyourresultslooklikethis?

Ofthetotalsample,50%reportedthattheywereRepublicans,42%wereDemocrats,5%wereIndependent,1%belongedtotheGreenParty,and3%hadnopartyaffiliation.

Willyouproduceaveragestoappearthisway?

Theaverageageoftherespondentsis56.4years.Themedianeducationallevelis13years.

Willyoucomparegroupsandreportsomethinglikethis?

Atotalof60%ofthemen,butonly20%ofthewomen,wereRepublicans.

Respondentsdonotdiffersignificantlyinsatisfactionwiththepresentgovernment.

Willyoulookforrelationshipssuchasthis?

Thesurveyfoundnoassociationbetweenhowliberalorconservativepeoplewereandtheireducationalattainments.

Highschoolgraduateswhowere30yearsofageorolderweresignificantlymorelikelytovoteinthelastelectionthanwereolder,lesseducatedrespondents.

Willyoulookforchangesovertime?

Since2007,statisticallysignificantdifferenceshavebeenfoundinthenumberofmenparticipatingin2ormorehoursofchildcareperday.

PilotTestingApilottestisatryout,anditspurposeistohelpproduceasurveyformthatisusableandthatwillprovideyouwiththeinformationyouneed.Allsurveysmustbepilottestedbeforebeingputintopractice.Self-administeredquestionnairesdependheavilyontheclarityoftheirlanguage(itdoesnotmatterifitisawrittenoronlinequestionnaire),andpilottestingquicklyrevealswhetherpeopleunderstandthedirectionsyouhaveprovidedandiftheycananswerthesurveyquestions.Apilottestofa

face-to-faceinterviewalsowilltellyouaboutinterviewers.Cantheyfollowtheinterviewformeasily?Arethespacesonprintedsurveyslargeenoughforrecordingresponses?Dointerviewersknowwhattodoifthecomputer“freezes”whiletheyareinthemidstofacomputer-assistedinterview?Doestherespondentunderstandhowtomovebackandforwardthroughanonlinesurvey?Pilottestsalsocantellyouhowmuchtimeittakestocompletethesurvey.

Testinghelpsmakethesurveyrunsmoothly.Wheneverpossible,youshouldtrytoduplicatetheenvironmentinwhichthesurveyistotakeplace.Thatmightmeanobtainingpermissionfrompeoplejusttobeinthetryouts,butnotinthesurvey,althoughtheyareeligibleforfullparticipation.

ResponseRateThesurveyorwantseveryonewhoiseligibletorespondtoallquestions.Pilottestinghelpsimprovetheresponseratebecauseitcaneliminateseverepotentialsourcesofdifficulty,suchaspoorlywordedquestionsandnoplacetorecordanswersonprintedquestionnaires.Furthermore,iftheentiresetofsurveyproceduresiscarefullytested,thenthis,too,canhelptheresponserate.Beforeyoudoatelephoneinterview,askthesequestions:Doyouhaveavailableacurrentlistoftelephonenumbers?Areyouwillingtomaketelephonecallsatthetimethesurveyrespondentsareavailable?Doyouhaveaplanforreachingrespondentswhodonotreturncallsleftonvoicemail?Foronlinesurveysaskthefollowing:Doyouhaveavailableacurrentlistofemailaddresses?Doyouknowhowtoensureprivacyofresponses?Otherwaysofensuringgoodresponseratesexist,regardlessofsurveytype,suchaskeepingsurveysshortandsimpleandprovidingincentivesforparticipating.

Howhighshouldtheresponseratebe?Ifyouareconductingalarge,complexsurvey,youwillwanttousestatisticalprocedurestoanswerthisquestion.Ifyoursurveyisrelativelysimple(say,apoolofteachersinaschoolornursesinthreehospitals),thenyouhavetodecidehowmanypeopleyouwillneedfortheresultstobebelievable.If20peopleareeligibleforcompletingamailed,self-administeredquestionnaireandonly10respond,youmayfeeldifferentfromthewayyouwillfeelif,atanothertime,200of400respond.Bothsurveyshavea50%responserate,butreportingontheviewsof10of20peoplemayappeartobelessconvincingthantellingabout200of400.Exceptwhendonestatistically,thedesiredresponseratetendstobeentirelysubjective,andthegeneralruleis“higherisbetter.”

ReportingResultsSurveyresultsarereporteddailyontheInternetandTVandinnewspapersandmagazines.Tomany,asurveyisapollusuallyofsome,butnotall,peopleaboutanissueofimmediatepolitical,social,oreconomicconcern.Surveyresultstypicallylooklikethis.

Example:TheLookofSurveyResults(1)Question:Iftheelectionwereheldtoday,wouldyouvoteforCandidateX?

Example:TheLookofSurveyResults(2)

Source:Takenfromwww.questionpro.com.

Example:TheLookofSurveyResults(3)

http://www.questionpro.com

Source:ASurveyofApplicants—SenateCommittee.

MoreinformationonstandarderrorsisfoundinChapter5,andmoreinformationonconfidenceintervalsandstandarddeviationsisprovidedinChapter6.

Togetresultssuchastheserequiresmanysteps,andallsurveysfollowthem:

Decidingonthetypeofsurvey(mailedoronlinequestionnaire;telephoneorface-to-faceinterviews)Selectingthesurvey’scontentandwritingquestionsandtryingouttheformDecidingwhoshouldparticipate(Everyone?Asampleofpeople?)andhowoften(Justonce?Eachyearfor5years?)Administeringthesurvey(Whoshouldconducttheinterview?Bywhenmusttheonlinequestionnairebesubmitted?)Processingthedata(Howwilldatabeentered:manuallyorelectronicallyfromsurveytodatabase?)Analyzingandinterpretingtheresults(Wastheresponserategoodenough?Whatdothenumbersordifferencesbetweenpeoplemean?JusthowdopeoplefeelaboutCandidateX?Haveopinionschangedovertime?)Reportingtheresultsorallyorinwritingusingtext,charts,tables,andgraphs(Whoistheaudience?Howlongshouldthereportbe?Whichdatashouldbepresented?)

Nocrediblesurveycanomitanysinglestep,although,dependingonitspurposesandresources,somestepswillreceivemoreemphasisinanygivensurveythaninanother.

TheFriendlyCompetitionHowdoyouchoosebetweenself-administeredquestionnairesandinterviews?Isamailedoranonline

surveybetter?Whenisaninterviewthebestchoice?Herearesomecriteriaforselectingamongthedifferentsurveytypes.

ReliabilityandValidityAreliablesurveyresultsinconsistentinformation.Avalidsurveyproducesaccurateinformation.Reliableandvalidsurveysareobtainedbymakingsurethedefinitionsandmodelsyouusetoselectquestionsaregroundedintheoryorexperience.Nosinglesurveytypestartsoutwithbetterreliabilityandvaliditythananother.Choosethesurveymethodthatismostpreciseandaccurateforyourspecificpurposes.Forexample,ifyouareworriedthatthepeopleyouaresurveyingcannotreadwell,anoral(face-to-faceortelephone)interviewislikelytoproducefarbetterresultsthanawrittenone.

Focusgroupsandpilottestshelpyoudecidewhichtypetouseandwhetheryouhavedoneagoodjobofdesigningthesurveyandmakingituser-friendly.Respondentsorsurveyadministrators(thepeoplewhodotheinterviewingorhandoutthequestionnaires)whohavetroublewiththesurveywilluseitincorrectly,introducingbias,andthatinturnreducestheaccuracyoftheresults.Awelldesigned,easy-to-usesurveyalwayscontributestoreliabilityandvalidity.

UsefulnessorCredibilityofResultsTheresultswillbeusefuliftheyarevalidandifthesurveymethodisonethatusersacceptascorrect.Findoutbeforeyoustartwhichmethodistheonepeoplewant.Sometimesthepeoplewhowillusetheresultshavestrongpreferences.

CostsThisreferstothefinancialburdenofdevelopingandadministeringeachtypeofsurvey.Thecostsassociatedwithwrittenquestionnaires(on-siteandmailed)includepaper,reproduction,andincentives.Mailedquestionnairesrequireanup-to-dateaddresslist(whichyoumayhavetopurchase),postage,andenvelopes.Sometimesyouhaveseveralfollow-upmailings,addingtothecosts.

Thecostsofface-to-faceandtelephoneinterviewsincludephonechargesor,inlargesurveys,thecostsofpurchasingatelephonesystemandpayingformiscalledandout-of-datetelephonenumbersaswellashang-ups.Youalsoneedtopayforwritingascriptfortheinterviewer,trainingtheinterviewers,monitoringthequalityoftheinterviews,andprovidingincentivestorespondents.

Onlinesurveysrequireextensivedevelopmentandtesting.Onewaytosaveondevelopmentistopurchaseuser-friendlyonlinecommercialsurveyprograms.Youwillhavetolearnhowtousethedevelopers’software,however,andthiscanbetime-consumingandcostly.Andeventhebestsoftwarecannotmakeupforasloppysurvey.

TheSpecialCaseofOnlineSurveysOnlinesurveysareself-administeredquestionnaires.Respondentscompleteonlinesurveysonlaptops,desktops,notebooks,tablets,andcellormobilephones.Surveyorslikeonlinesurveys,andrespondents

arebecomingusedtothem.Surveyorslikeonlinesurveysbecausetheycaneasilyreachverylargenumbersofpeopleacrosstheworldandbecauseonlinesurveysoftwareisaccessibleandrelativelyinexpensive.

Onlinesurveyssometimesrunintotroublebecausesurveyorsdonotplanaheadadequately.Surveynoticesaresentouttounsuspectingrespondents,whothendeletethemasspam.Orthesurveyissenttoeveryoneonalistofemailaddresses,andmanyoftheaddressesareincorrect.Otherproblemsincludefailuretoprovideincentivestooverloadedrespondentswhoareoverwhelmedwithsurveyrequestsortoomanyorhard-to-understandquestions.Aswithanyothersurveytype,thesurveyorhasmanycarefuldecisionstomakeaboutthecontentandformatofthequestionsandresponses,thesamplesize,whowillbeincludedinthesurveyandwhowillbeexcluded,andhowtoguaranteethattheresultsareaccurateandsecure.Onlinesoftwarecannotmakeupforapoorlydesignedandimplementedsurveythatisnotusedappropriately.

Hereisachecklisttouseindecidingwhentouseonlinesurveys.

ChecklistforDecidingtheBestUsesforOnlineSurveysThesurveyneedswidegeographiccoverage.Youarehopingtoreachaverylargenumberofpeople.Youhaveaccesstovalidemailaddressesandpermissiontousethem.Unsolicitedsurveysmayappearasspamorjunkmail.Youoryouragencyorinstitutionconductssurveysrelativelyfrequentlysothatanyinvestmentintechnologicalandsoftwareinfrastructureisworthwhile.Developingsoftwaretocreatesurveyquestionscanbecostly.Commercialcompaniesaboundwhocanhelpdesignandimplementsurveysandanalyzedata.That’stheeasypart.Thereisalwaysalearningcurve,however,andlearningnewsoftwaretakestime,andtimeusesfinancialresources.Youhavedesignedsomesurveyquestionstobeansweredbyallrespondents,butotherquestionswillonlybeansweredbysomerespondents.Forinstance,ifyouwanttoexplorepeople’sexercisehabits,butnotallpeoplereportthattheyexercise,youonlywanttoasktheexercisersabouttheiractivities.Onlinesurveyscanbeprogrammedtodistinguishbetweenwhoistoanswerquestionsandwhoistoskipthem.Thissavesthesurveyortheneedtoresorttocomplexinstructions.Youplantousemultimediasuchasvideoclipsandvoice-overindescribinganewidea,product,orservice.Youneedresultsinahurry.Onlinesurveyscanproducereal-timedataaseachpersoncompletesthequestionnaire.Itisimportanttorememberthatthestatisticalresultsproducedbymostsurveysoftwarepackagesarerelativelysimple:numbers,averages,andsoon.Thegraphicsthatareproducedarealsosimpleandincludepiechartsandlinegraphics.Morecomplexanalysesandreportstakemoretimeandexpertise.Youwanttocomparesurveydataovertime.Onlinesurveydatacaneasilybestoredinadatabaseforfutureuse.Newsurveydatacanbeaddedtothedatabasefromtimetotimefromthesameorfromdifferentgroupsofpeople,andyoucancomparedataovertimeforoneormoregroupsofrespondents.Youplantostudysurveybehavior.Withonlinesurveys,thedateandtimeofsurveycompletionandthetimespentoneachquestionareknown.Thisinformationcanbeusedtoseeiftherearedifferencesamongpeopleinhowmuchtimetheyneedtocompletesomeorallofthesurvey.Thedatacanbeusedtohelpyouimprovethesurveyquestions.

Atleastfourpotentialconcernsaccompanymostwebsurveys.Theyarethefollowing:

1.Multipleaddresses.Manypeoplehavemanyemailaddressesorpointsofonlinecontactthroughsocialmedia.Althoughsomepotentialrespondentsmayconsolidatetheiraddressessothattheycanviewmessagesfromallsourceswhenevertheycheck,othersdonot.Somepeopleonlycheckoneortwositesfrequently,andotherschecktheiremailinfrequently.Becertainthattheaddressesyouhaveareaccurateandcommonlyused.

2.Privacyandconfidentiality.Manypeoplearesuspiciousofsendinginformationovertheweb.

Surveyorsmustlearnhowtoguardrespondents’privacyandconvincinglycommunicatethestrengthoftheireffortstorespondents.Itisimportanttonotethatnoteveryoneknowsiftheircomputerortabletissecureorhowtomaximizesecurity,noraretheyawareofwhohasaccesstotheirbrowsinghistory.Thesurveyor,ofcourse,doesnothavecontroloverprivacylapsesthatmayoccurbecauseofapoorlysecureddevice.

3.Identityoftherespondents.Thesurveyorhasnorealwayofknowingwhoisactuallycompletingthequestionnaire.

4.Technologicaldifferences.RespondentsvaryintheirInternetconnectionsandbrowsersandmaysettheirowntextsizeandcolorsettings.Somesurveysdonotpermituseofthebackfunctiononthebrowserandrelyinsteadonaspecialbackarrow.Anydeviationfromexpectationmaybeconfusingtorespondents,andtheymaygiveup.Thesurveyshouldbetestedonallcommonlyusedbrowsers.Ifnecessary,respondentsshouldbeinstructedonhowtogobackandforwardinthesurvey.Computers(desktops,laptops,tablets)losetheirconnections,anditisagoodideatosetupasystemsothat,incaseoflossofpoweroraconnection,therespondentswillbeabletoreturntothesurveywithoutlosingtheirplace.

Manywebsurveysaredonethroughonlinemerchantswhosellsoftwareandsurveyprocessingservices.Manyfirmsdopracticallyeverythingforthesurveyor.Theywillsendoutthesurvey,trackrespondents,analyzedata,andproducereports.Theydon’tusuallycreatethesurvey,buttheymayprovidereferralstoconsultantswhodo.

Ifyoudecidetopurchaseonlinesurveysoftware,bepreparedtospenduptoaweeklearninghowtouseit.Mostvendorsprovideexcellenttutorialsinhowtoselectquestiontypes,numberpagesandquestions,anddecideonfeaturessuchasbackgroundcolorsandlogos.Itisveryimportantthatyoureadthefineprintregardinghowmuchassistanceyouwillhavefromthesurveyvendor.

Surveyorsaretemptedtorelyontheirvendorstoupdateandmaintainthesurveyandthesurveydata.Butservicecontractstendtobelimited,andsotheseguidelinesshouldbefollowed.

Youcangoonlineforinformationonhowtoevaluatemerchants.Gotoanysearchengineandusethesekeywords:“compareonlinesurveyservices.”

GuidelinesforSurveyorsWhoWorkWithCommercialSurveyCompanies

Downloadyourdatafrequently,andstoreyourfilessafely.Youraccesstoyourdatamaydisappearwhenyoudiscontinueyourservice.Createofflinecopiesofeveryonlinesurveysothatyoucanrecreatetheinstrumentinapaperandpencilformatoruseitwithanotheronlineservice.Makesurethatyoucansavethesurveyonyourcomputer.Bewaryofthecompany’slistofsuggestedsurveyquestions.Aretheyappropriateforyourrespondents?Whatevidencehasbeenprovidedthattheyhaveeverbeenusedwithanyone?Bywhomhavetheybeenused?Makecertainthatyouknowwhoownsthesurveydata.Doyouownthedata,ordoesthevendor?Canthevendorsellthedata?Useitforpurposesunrelatedtoyoursurvey?Whathappenstotherespondents’emailaddresses,ifyouusethem,asiscommontoinvitepeopletotakethesurvey?Aretheresponsescollectedoversecured,encryptedconnections?Whereistheinformationsystem’sinfrastructure(servers,networkingequipment)located?Whoownsandmanagestheequipmentinthedatacenters?Arethedatacentersstaffedandmonitoredallday,everyday?Whohasaccesstothedatacenters?Theanswerstothesequestionsshouldbereadilyavailableonthevendor’swebsite.Lookforthevendor’sprivacypolicyand

readit.

TheSpecialCaseofCellorMobilePhonesManylargesurveysthathavehistoricallyreliedonlandlinesamplingarenowalsousingcellphonesamples.Unfortunately,surveyorsarefindingthatcellphonesurveyscanbemoreexpensivethantheythought.U.S.federalregulationsprohibittheuseofautomaticdialingtocellphonelines,andhanddialingincreasescosts.Also,screeningforeligiblerespondentscanraisethecosts.Atthepresenttime,manyyoungerpeopleusecellphones,butolderpeoplepreferlandlines.Reachinganeligiblerespondentsuchastheheadofahouseholdoranolderadultcanmeanextraeffortandexpense.

Cellphonesurveysarealsocomplicatedbythefactthattheyareattachedtoindividualsratherthangeography,soanadditionalscreenmustbeaddedtoensurethattherespondentlivesintheareaforwhichthesurveyorwantstocollectdata.Theprocessbecomesevenmorecomplicatedwhenyouconsiderthatsomepeoplehavelandlines,somehavelandlinesandcellphones,andsomeonlyhavecells.Noonereallyknowsifthesegroupsdifferfromoneanother,andiftherearedifferences,whethertheyaffectthesurvey’sresults.

Cellphonesurveysalmostalwaysrequireincentivesforparticipationbecausetherespondentotherwisepaysforthecalldirectlyorindirectly.

MakingtheDecisionTohelpyoudecideamongthedifferenttypesofsurveysthatmayberelevanttoyourneeds,Table1.2comparestheadvantagesanddisadvantagesofthemajorsurveytypesandremindsyouoftheirspecialneedsandcosts.

ASurveyContinuum:FromSpecifictoGeneralUseSurveyshavebecomeamajormeansofcollectingdatatoanswerquestionsabouthealthandsocial,economic,andpoliticallife.Howextensiveandscientificmustasurveybe?

Comparethesetwosurveys.

Example:SurveyWithaSpecificUseThedirectorsoftheNeighborhoodHalfwayHouseswanttoprovideservicesthatareappropriateforresidents.Atpresent,manycomplaintshavearisenoverthelackofadequatefitnessfacilities.Asurveywillbeconductedtopollthe5healthcareproviders,100residents,and10full-andpart-timestafftofindoutwhatfacilitiesaredesirableandaffordable.

Example:SurveyWithaGeneralUseTheCountyHealthDepartmentisconcernedwiththeeffectivenessofits10halfwayhouses.Together,the10houseshave20,000residentsand220full-andpart-timestaff.TheCountyHealthDepartmenthasnegotiatedarrangementsforhealthcareservicesfromanumberofprovidersinthepublicandprivatesectors.Aspartofitseffectivenessstudy,theCountyHealthDepartmentissurveyingarandomsampleofresidents,staff,andprovidersatallhouses.NextDoorCountyisinterestedinadoptingtheCountyHealthDepartment’shalfwayhousemodelandisanxiouslywaitingfortheresultsofthesurveyandevaluation.

Thejustificationforthefirstsurveyisonehalfwayhouse’sconcernwithitsownneeds.ThereasonforthesecondistheCountyHealthDepartment’sinterestintheeffectivenessofallitshalfwayhouses.Also,NextDoorCountyisinterestedinthesurvey’sresults.Survey1,withitslimitedimpact,canberelativelyinformalinitsmethods.Survey2,incontrast,mustberigorousinitssamplingplan,questionnaireconstruction,anddataanalysisandinterpretation.Survey1isconcernedprimarilywithusefulness.

Survey2alsoisconcernedwithvalidityandgeneralizability:Ifadaptedinanotherplace(NextDoorCounty),willtheCountyHealthDepartment’shalfwayhousemodelbeequallyeffective?

Eachtimeyoudoasurvey,youmustevaluatewhereitspurposesfallonacontinuumthatgoesfromspecifictogeneraluse.Youhavesomeleewaywithasurveydesignedtomeetspecificneeds.Allsurveysthataimtobegeneralizableintheirfindingsmustbeconductedwithrigor.

Ethics,Privacy,andConfidentialitySomepeoplehavebecomesuspiciousofsurveys.Theyfearthattheinformationtheyprovidewillbeusedinappropriately.Manytechniquesexistforprotectingeachperson’sprivacyandensuringthatinformationwillbeusedonlywiththeperson’sknowledgeandforclearlystatedpurposes.Thesurveyorneedstoreassurepotentialrespondentsthatthesetechniqueshavebeenincorporatedintoeachsurvey.Itistheethical—rightthing—todo.

Allcompletedprintedorwrittensurveysshouldbekeptinlockedfiles,andonlyalimitednumberofstaffshouldhaveaccesstothemonaneed-to-knowbasis.Attheconclusionofdataanalysis,thesurveysshouldbeshredded.Furthermore,youcanseparateidentifyinginformation(e.g.,names,birthdates,SocialSecurityorgovernment-issuedidentificationnumbers)fromsurveyresponsesbyassigningcodesto

individualsandusingthecodestolinkthemtotheirresponses.Onlinesurveytakerscanbepermittedtoassigntheirownusernamesandpasswordswhenloggingintotakeasurvey.

Theuseofsurveysandconcernforethicalissuesarecompletelyinterwoven.Surveysareconductedbecauseoftheneedtoknow;ethicalconsiderationsprotecttheindividual’srighttoprivacyorevenanonymity.

IfyoursurveyisforapublicorprivateagencythatisreceivingU.S.governmentfunds,youshouldknowthatthefederalgovernmenthasspecifiedthelegaldimensionsofinformedconsent,privacy,andconfidentiality.Thesedimensionsincludethefollowing:

AfairexplanationoftheprocedurestobefollowedandtheirpurposesAdescriptionofanyrisksandbenefitsAnoffertoansweranyinquiriesAninstructionthatthepersonisfreetowithdrawconsentanddiscontinueparticipationwithoutprejudice

Confidentialityisprotectedbythe“ProtectionofHumanSubjects”guidelinesoftheCodeofFederalRegulationssometimescalledtheCommonRules.Confidentialityreferstothesafeguardingofanyinformationaboutonepersonthatisknownbyanother.Asurveyorwhohasthenamesandaddressesofpeople,evenincodeorwithoutidentification,“deidentified,”maynotusethisinformationtorevealidentities.Inmanysurveys,confidentialityisarealconcernbecausecompleteanonymityispracticallyimpossible.Acodenumber,anemailaddress,anInternetServiceProvider(ISP)address,orevensometimesjustaziporpostalcodemayleadtothesurveyrespondent’sidentity.

Ifyouworkforaprivateagency,organization,orbusiness,youshouldchecktherulesofinformedconsentandconfidentiality.Isthereahumansubjects’protectioncommitteeorInstitutionalReviewBoard(IRB)whoseapprovalyoumustget?Ifyouareastudent,checktoseewhetheryoucanaskthequestionsyouareplanning.Also,youmaybepartofalargerprojectthathasalreadyreceivedapprovalforitsactivitiesaslongasitconformstocertainstandards—amongthem,theinformedconsentofrespondents.

InformedConsentTheconsentformgivespotentialrespondentssufficientwritteninformationtodecidewhethertocompleteasurvey.Hereisalistofcontentstoincludeinaninformed-consentform:

ContentsofanInformed-ConsentFormAtitlesuchas“ConsenttoParticipateinSurvey.”Thenameofthesurvey.Thepurposeofthesurvey.Procedurestobefollowed,includingwherethesurveywilltakeplaceanditsduration.Potentialrisksanddiscomforts,includingansweringquestionsthatarepersonalorbeinginaclosedroomfor2hours.Potentialbenefitstorespondentsandsociety,includingnewknowledgeorbetterinformationtodevelopprogramsorpolicies.Sometimesthebenefitsarenotyetreallyknown.Paymentforparticipation.Sayhowmuchparticipantswillbepaid;ifnopaymentisprovided,sayso.Confidentiality.Iftherespondent’snameistobekeptconfidential,describecodingprocedures,whowillhaveaccesstothesurveys,andwherethecompletedsurveyswillbekept.Ifinformationistobesharedwithanyone,statewithwhom.Youmayberequiredbylawtorevealsurveyresults.

Participationandwithdrawal.Cantheparticipantswithdrawatanytime?Whathappenstothemiftheydo?(Forexample,dotheystillretainanyincentives?Willtheystillreceivethesameeducation,socialbenefits,orhealthcaretheycamefor?)Identificationofsurveyors.Whoshouldbecalledifquestionsariseaboutthesurvey?

TheInternetandEthicalSurveysAnonlinesurveyinvolvesawebofcomputersthatinteractwithoneanother.Communicationstakeplacebetweenthesurveyorandtherespondent,therespondentandthewebserver,andthewebserverandthesurveyor.Securitybreachesarepossibleanywherewithinthewebunlessyouputprotectionsinplace.

CommunicationBetweentheSurveyorandtheRespondentItisnotuncommonforasurveyortocontactarespondentbyemail.TheemailwilldiscussthesurveyandinvitetheparticipanttoclickonaURLorpasteitintoabrowsersuchasMozilla’sFirefox,Google’sChrome,Apple’sSafari,orMicrosoft’sInternetExplorer.Unfortunately,emailisnotalwayssecureorprivate.Manypeopleareunawareofwhethertheircomputersaresecureorevenhowtosecurethem.Emailprogramsmaintainedbyemployersoftenarenotprivate.Ifpeopledonotlogofforarecarelessaboutpasswords,theirprivacycanbecompromisedeasily.Also,inadequatepasswordsareeasytocrack.Ifyouconductasurveythatrequirespeopletouseapassword,youmustensurethatthepasswordsetupissecure.

CommunicationBetweentheRespondentandtheWebsiteWhenarespondententerssensitivedataintheblankspacesofaweb-basedquestionnaire,itissimilartoashopperprovidingacreditcardnumberwhenshoppingonline.OnlinemerchantsuseaSecureSocketsLayer(SSL)protocolthatallowssecurecommunicationsacrosstheInternet.AnSSLprotocolencrypts(convertsintocode)theuser’ssurveyinput,anditdecryptsitwhenitarrivesatthewebsite.Manypotentialsurveyrespondentsarebecomingawareofhoweasilytheirresponsescanbeinterceptedunlesstheyaresecured,andwithoutguaranteesthatresponsesareencrypted,someofthemmayrefusetotakethesurvey.YoumustdecideinadvancewhethertouseSSLandhowtoexplainyoursecuritychoicestorespondents.

CommunicationBetweentheWebsiteandtheResearcherSensitiveidentifiabledataneedtobeprotectedintransitbyusingeitheranSSLprotocolorasecurefiletransferprotocol.

DataProtectionSomepeoplearereluctanttocompleteonlinesurveysorevenconnecttosurveysitesforfearthattheirprivacywillbecompromised.Alldatabasesstoringsensitiveandidentifiableinformationmustbeprotected,regardlessofwhethertheyarecreatedandmaintainedbycommercialfirmsorbyindividuals.Encryptingthedatabasesprobablyprovidesthemostsecurity.

Allreputableorganizationsdeveloporadaptrulesforreassuringrespondentsthatprivacywillberespected.Hereisaminimumsetofrulesforaprivacypolicy:

ThefollowingistheCentersforDiseaseControl’sprivacypolicy(http://cdc.gov/privacy.html).AlthoughtheCDC’ssiteisusedformanypurposes(notjustforsurveysandsurveyreports),itsprivacypolicyiseasytoreadandillustrateshowtocomplywiththeminimumsetofcriteriaforaprivacypolicy.

http://cdc.gov/privacy.html

Lookatthefollowingexcerptfromaprivacystatement.Thestatementcomesfromaverylargecorporationthatconductssurveys.Asyoucansee,thecompanyistruthfulaboutthepotentialforothercompaniestotrackcustomers’activities.However,consumersareleftwiththeobligationto(1)beawarethatunwantedcookiesmaybeplacedontheirharddriveand(2)iftheyprefer,todosomethingaboutitbycontactingtheprivacyofficer.

Thisexcerptraisesseveralquestions:Willtherespondentsactuallyknowifcookiesareontheirharddrive?Howdoestherespondentgetintouchwiththeprivacyofficer?Informationisavailableinthe“ContactUs”portionofthesite,buttherespondenthastolookforit.Itmakessensethatthepublicisincreasinglysuspiciousofonlinesurveysandhowtheirdataareused.

Youcanhelpavoidsomeoftheseproblemsbybeingcertainyouhaveconsideredallthepitfallsofsendingsurveysandsurveyinformationintocyberspace.

IfyouplantousetheInternet(includingemail)to(1)communicatewithstudyparticipantsor(2)sendparticipantinformationtoacollaboratororcontractor,youshouldbeabletocompletethefollowingquestionnaireformaintaininganethicallysoundonlinesurvey.

Eachsurveyhasdifferentlimitsonwhatitneedstocollectandfromwhom.Somesurveysamplesaremorevulnerablethanothersandneeddifferentsafeguards.Thefollowingisaninformed-consentformtypicalofonethatcouldbeusedinanonlinesurveyofteachersinalargeschooldistrict.Thesurvey’spurposeistoidentifyneedsforimprovementintheworkplace.

Somelargeinstitutionsandcompanieshaveethicsboardsandprivacyofficerswhocanhelpyouensureanethicalsurvey.Manysurveyorsandsurveycompanies,however,arenottechnicallysophisticatedregardingprivacy,noraretheytrainedinonlinesurveyethics.Youcanlearnmoreaboutethicalsurvey

researchbygoingonlinetotheNationalInstitutesofHealth’sguidelinesonethicalresearch(http://grants.nih.gov/grants/policy/hs/ethical_guidelines.htm)andtheCollaborativeInstitutionalTrainingInitiative,whichprovidestraininginethicalresearchwithhumansubjects(https://www.citiprogram.org/Default.asp).

ExampleQuestionnaire:MaintaininganEthicallySoundOnlineSurvey

1. DescribethemeasuresthatwillbetakentoensurethatthewebserverhostingtheInternetsiteisprotected.Inthedescription,provideinformationonphysicalsecurity,firewalls,softwarepatches/updates,andpenetrationdrills.

2. Ifapasswordorothersecureauthorizationmethodistobeusedtoallowaccesstothewebsite,askthesetwoquestions:Howwilluserpasswordsbedistributed?Howwillpasswordsandwebaccessbeterminated?

3. Iftheusersessionisencrypted,describethemethodofencryptionthatwillbeused.4. Explainwhowillhaveadministrativeaccesstodataonthewebserver.Givenames,studyroles,andorganizational

affiliations.5. Explainindetailtheadministrativesafeguardsputinplacetorestrictunauthorizedandunnecessaryaccess.6. Describehowtheinformationwillbeused.Willyougive,sell,ortransferinformationtoanyone?7. Givethenameandaddressoftheapplicationowner,thatis,thepersonsorpersonwhomaintainstheapplication.8. Ifemailisusedtocontactrespondents,describethemeasurestakentoassurerespondentsthatthecommunicationisfrom

anauthorizedperson.9. Ifrespondentsareaskedtocontactthesurveyorsusingemail,describehowtherespondentswillbeauthenticatedto

adequatelyensurethesourceoftheemailcommunication.10. Explainhowthestudyconsentformdescribesthepotentialriskstoprivacyassociatedwiththeuseofemail.11. Ifemailistobeusedtosendstudydatatoinvestigators,vendors,orothers,explainifandhowtheemailwillbeencrypted.12. Ifrespondentsaretosendyouattachmentsbyemail,tellthemiftheattachmentswillbeencryptedorpasswordprotected.13. Ifautomatedemailroutingsystemsareused,describethesecuritycontrolsthatwillbeinplace.Specifically,describethe

testinganddisasterrecoveryprocedures.14. Ifcontractorsorvendorshaveaccesstosurveyrespondents’personalidentifiableorconfidentialinformation,

describethelanguagethatisincludedinthecontracttoprotectrespondentprivacyanddescribethesecurityrequirementsthatwillbeprovidedtocontractorsorvendorswhoaredesigningorhostingweb-basedservicesfortheproject.

15. Givethenameofthepersononthesurveyprojectresponsibleforensuringthatthesurveyorganization’spoliciesandproceduresforconfidentialityandsecurityarefollowedforthisproject.Providehisorherprofessionalpositionandaffiliation.

16. Givethenameofthepersonresponsibleforthegeneralsecurityadministrationfortheinformationtechnologyassociatedwiththisparticularsurvey.Providehisorherprofessionalpositionandaffiliation.

Example:Informed-ConsentFormforanOnlineSurveyYourindividualresponsestosurveyquestionswillbekeptconfidentialbyTheSurveyProjectanditssurveycontractor,OnlineSystems,Inc.Confidentialdataareinformation,suchasanindividual’sorschool’sidentification,thatmaynotbereleasedoutsideofTheSurveyProject,exceptwithpermissionfromtherespondent.IndividualsmaygrantTheSurveyProjectpermissiontoreleaseconfidentialdatathatdescribethemselves.AnauthorizedrepresentativeofaSurveyProjectmemberschoolmaygrantTheSurveyProjectpermissiontoreleaseconfidentialdatathatdescribehisorherschool.[Comment:Thisdefinesanddescribesthelimitsofconfidentiality.]

OnlineSystems,Inc.willgenerateaggregatereportsthatcontainschoolwideanddepartmentalinformationtohelpyourschoolidentify,prioritize,andimplementimprovementsintheschoolworkplacethatwillincreasestudentengagement.Informationwillnotbereportedininstanceswhererespondentgroupscontainlessthanfiveindividuals.[Comment:Itmaybepossibletoidentifyindividualviewsinverysmallgroups.Thiswouldviolateprivacy.]Datafromopen-endedquestionswillbeprovidedtoyourschoolindeidentified,redactedform.OnlydeidentifiedrecordleveldatawillberetainedbyTheSurveyProject,andonlydeidentifiedaggregateanalyseswillbesharedinpublicationsandresearchpresentationswiththeacademiccommunity.[Howthedatawillbeused]TheSurveyProjectmayreleasedeidentifiedresponsestoindividualswhoagreetoprotectthedataandwhoagreetoTheSurveyProject’sconfidentialitypolicies.OnlineSystems,Inc.willstoredataonsecureserversandwilldestroyall

http://grants.nih.gov/grants/policy/hs/ethical_guidelines.htm

https://www.citiprogram.org/Default.asp

identifieddatawithin2yearsofsurveyadministration.Byparticipating,youwillbecontributingvaluableinformationtoyourschool.[Comment:Serverswillbesecured.Thevendormustdestroyidentifiabledatawithin2years.]TheSurveyProjectandOnlineSystems,Inc.havetakennumerousstepstoprotectparticipantsintheSurveyProject.EthicsBoardrequirementsrequirethatyouareinformedthatiftheinformationcollectedweretobecomepublicwithindividualidentificationitcouldprovepersonallyuncomfortable.[Comment:Thisisariskofparticipation.]

ThissurveyhasbeenreviewedandapprovedaccordingtoTheSurveyProject’spoliciesandprocedures.Bycontinuing,youacknowledgethatyouhavereadandunderstoodtheaboveinformationandagreetoparticipateinthissurvey.[Comment:Thisisanonlinesurvey,andtherespondentisnotaskedto“sign”toindicatewillingnesstoparticipate.Signingsoftwareisavailable,butmostsurveysacceptsurveycompletionasinformedconsent.]Ifyouhaveanyquestionsaboutthesurvey,contact....Ifyouhaveanyquestionsaboutyourrightsasaresearchparticipant,contact...[Comment:Whomtocontactwithquestions].

ChildrenandSurveyEthicsWhoisachild?Thedefinitionofachilddiffersacrossresearchjurisdictionssuchascountriesandindividualstates.IntheUnitedStates,achildusuallybecomesanadultby18yearsofage.Thatmeansthata17-year-oldintheUnitedStatesmaystillbeconsideredachild.IntheUnitedStates,theChildren’sOnlinePrivacyProtectionAct(COPPA)requiresverifiableconsentfromaparentorlegalguardianforinterviewingchildrenbelowtheageof13years.Therulesforethicalsurveysareoftenstricterforchildrenthanforadults.

Whendoingsurveyswithadults,thefirststepistomakesuretheyknowwhattheyaregettingintoandhowtogetoutofitiftheywantto.Thatmeanswithadults,thefirststepforparticipationistoprovidethemwiththemeansofprovidinginformedconsent.

Theinformedconsentmodelwithchildrenrequiresparentsorlegalguardianstoprovidepermissionfor(1)theirchildrentotakethesurveyand(2)thesurveyortocontactthechildren.Oncepermissionisobtained,thechildrenareaskedfortheirassenttobecomeparticipants.

IntheUnitedStates,federalregulationsspecifywhatmustbeincludedinanadultconsentprocess,andtheserequirementsalsoapplytotheparentalpermissionprocess.Therearenoregulationsthatrequirespecificelementsordefinethecontentorformatofthechildassentprocess.Thismeansthatsurveyorsareleftontheirowntopreparetheadultandthechilddocuments.

Researchaboutchildren’sdecision-makingskillssupportsthecommonpracticeofusingdifferentassentprocessesforchildrenandforadolescents,withthelevelofdisclosureincreasingaschildrengrowolder.Surveyorsshouldprobablyoffermatureadolescentsinformationaboutthesurveythatiscomparabletoinformationthattheyprovidetoadults.

Don’tforgettoconsiderthenationality,ethnicity,andsocioeconomicstatusofyourpotentialrespondentswhenputtingtogetherparentalpermissionandchildassentprocessesanddocuments.Also,rememberthatculturalassumptionsabouttherightsofchildrenvarywidely.Insomecountriesorsubgroupsitmaybeinappropriateandperhapsoffensivetoaskchildrentocompletequestionnaireson-oroffline.

Whengoingthroughtheassentprocess,makesurechildrenknowthefollowing:

Theyarebeingaskedtoparticipateinaresearchstudyorinalocalefforttounderstandmoreaboutaproblem(howtoimproveourschool;howtomaketheinternetsafer).Thepurposeofthesurvey(tofindouthowchildrenliketolearn;tounderstandhowmuchtime

childrenspendonsocialnetworks).Howmuchtimeisinvolvedinparticipating(nomorethan5minutes).Whatwillhappentothemiftheyagreetoparticipate(youwillanswerquestionsonline).Theforeseeablerisksand/ordiscomforts(youmayfindsomequestionshardtoanswer)andanybenefitstheymayexperience(youmayhelpotherchildrenyourage).Thattheycanasktheirparentsorsurveyoranyquestionstheyhaveaboutparticipating.Thattheirparticipationisvoluntary(entirelyuptothem)andthattheymaystopatanytime.Thattheirfather/mother/guardianknowstheyhavebeenaskedtobeapartofthestudy.

Hereisasampleassentform,onethatismostappropriateforchildrenbetween10and15.

Example:ChildAssentFormHello,

Wearedoingastudytolearnaboutpeoplewhoarehonestandpeoplewhoarenothonest.Weareaskingyoutohelpbecausewedon’tknowmuchaboutwhetherkidsyouragecomeacrossmostlyhonestpeopleormostlydishonestpeople.Wewillusethisinformationtoeducateparentsandteacherstobecomemoreunderstandingoftheworldtheirchildrenlivein.

Ifyouagreetobeinourstudy,wearegoingtoaskyousomequestionsaboutpeople.Wewanttoknowifyouthinktheyareusuallyhonestornot.Forexample,wewillaskyouifyourfriends,peopleyouknow,yourteachers,andfamilymembersareusuallyhonestorusuallynothonest.Wewillnottellyouranswerstoanyone.

Youcanaskquestionsaboutthisstudyanytime.Ifyoudon’twanttoansweraquestion,youdonothaveto.Ifyouwanttostopansweringquestions,youcanleavethestudy.Absolutelynothingwillhappentoyouifyoudecidetoleavethestudy.Noonewillbeupset,andwewillnottellanyone.

Ifyoudon’twanttobeinthisstudy,donotsignthispaper.Beinginthisstudyisuptoyou.

Yoursignature:

Signatureofpersonobtainingassent:

InternationalSurveysIfyouareinvolvedinaninternationalsurvey,keepinmindthatthecontentofaquestionnairereflectsnotonlythelanguageoftheoriginatingcountrybutalsothestandards,expectations,values,andpreoccupationsofthesurveyorsandanylaypeopleinvolvedinhelpingtodevelopandtestthequestions.Linguistsandanthropologistsoftenpointoutthatlanguageandculturearenotthesame.Forexample,bothEnglandandtheUnitedStatesareprimarilyEnglish-speakingcountries,buttheircitizensdonotnecessarilysharethesamevalueswithrespecttoeducation,health,andsocialwell-being.

ThinkaboutthewordfamilyinEnglish.ManyAmericans,forexample,interpretfamilytoincludeonlyrelativeslikemotherorsisterormaybeauntanduncle.Butinothercultures,familyincludesamuchwidergroup.Aquestionsuchas“Hasanyoneinyourfamilyhadheartdisease?”requirestheinternationalsurveyortomakecertainthattheresponsechoicesareinclusiveandmeaningfultotherespondents.Anotheroptionistobespecificandask:Haveanyofthesepeopleinyourfamilyhadheartdisease(mother,father,grandmother,grandfather)?

English-speakingsurveyorsmayfindthatsometimesitisimpossibletofindequivalenttranslationsinlanguagessuchasArabic,Cantonese,Punjabi,orSwahili.Theselanguageshavedifferentrootsfrom

English.Theterms“checkup”and“Papsmear,”forexample,havenoconceptualequivalentinanyChineselanguage.

Ethicalresearchininternationalsettingsmeansthatculturallyappropriateproceduresmustbefollowedtoprotectsurveyparticipants.ManyIRBsandethicsboardsrequiresurveyorstosubmitproofthatthecustomsandlanguageoftheparticipatingcountrieshavebeenrespected.

Considerthisifyouaredoinganinternationalsurvey:

1. Besurethesurveyisavailableintheoriginallanguage(English)andintranslation.Besureyoucanprovideadetailedexplanationofthetranslationprocesstojustifyitsappropriatenessforthecountryinwhichitistobeused.

2. Ifthesurveyincludeschildren,youneedtobeabletoverifytheageatwhichparticipantshavetheabilitytoconsentorassenttoparticipationinsurveys.

3. Iflocalcustomsandregulationsaresuchthatactiveparentalpermissionwouldbeculturallyinappropriate,thesurveyorshouldcompileproofthatsuchpermissionisnotculturallyappropriate.Examplesofsuchproofwouldbedocumentswithspecificregulations(inEnglishandcertifiedtobeaccurate)thatindicatethatsuchpermissionisnotrequiredoranofficialletterfromarankingofficialinthecountryofinterestindicatingthatsuchpermissionisnotculturallyappropriate

4. Ifthesurveyoriseligibleforawaiverofactiveparentalpermission,heorsheshouldbepreparedtoprovideparentswithaletterdescribingthestudy.Thelettershouldbewrittenintheparent’slanguageandatanappropriateliteracylevel.

5. Thesurveyoralsoneedstogetlettersofagreementfromtheappropriateofficials(governmentofficials,schoolofficials,communityofficials)thatstatethatthesurveyitselfhasbeenreviewedandisacceptable.Besurethattheletterisonletterheadandhasanoriginalsignature.

FormalStandardsforSurveyEthicsTwomajorsurveyresearchorganizationscanprovideyouwithstandardsforconductingsurveyresearchtocoverallsituations.TheAmericanAssociationforPublicOpinionResearch(AAPOR)hasdevelopedacodetodescribetheobligationsthatallprofessionalshavetoupholdthecredibilityofsurveyandpublicopinionresearch.ToreadthecodeandtogetmanyotherresourcesincludinghowtoworkwithanIRB,gotohttp://www.aapor.org/AAPORKentico/Standards-Ethics.aspx.

AAPORalsoprovidesalistofsurveypracticesthatitcondemns.Thisincludesrequiringamonetarypaymentorsolicitingmonetarycontributionsfrommembersofthepublicaspartofaresearchprocess,offeringproductsorservicesforsale,usingparticipantcontactsasameansofgeneratingsalesleads,andrevealingtheidentityofindividualrespondents.

TheCouncilofAmericanSurveyResearchOrganizations(CASRO)haspreparedaCodeofStandardsandEthicsforSurveyResearch(http://www.casro.org/).CASROwasestablishedtorepresenttheU.S.researchindustryandorganizationsengagedintheconduct,support,oreducationofmarket,opinion,andsocialresearch,oftendescribedasdataanalytics,consumerinsights,orbusinessintelligence.CASRO’sethicsdocumentcoverstopicslikeconfidentiality,privacy,theavoidanceofharassment,internetresearch,responsibilitiestoclientsandresponsibilitiestooutsidecontractorsandinterviewers.Inthepreambletothecode,CASROstatesthatrespondentsshouldbewillingparticipantsinsurveyresearch,appropriatelyinformedaboutthesurvey’sintentionsandhowtheirpersonalinformationandsurvey

http://www.aapor.org/AAPORKentico/Standards-Ethics.aspx

http://www.casro.org/

responseswillbeusedandprotected,sufficientlysatisfiedwiththeirsurveyexperience,andwillingtoparticipateagaininsurveyresearch.

SummingUpSurveysareinformationcollectionmethodsusedtodescribe,compare,orexplainindividualandsocietalknowledge,feelings,values,preferences,andbehavior.Surveysarebestwhenyouneedinformationdirectlyfrompeopleaboutwhattheybelieve,know,andthink.Asurveycanbeaself-administeredquestionnairethatsomeonefillsoutaloneorwithassistance.Self-administeredquestionnairescantaketheformofwrittenoronlinesurveys.Writtensurveysmaybecompletedbymailoronlocation.Asurveycanbeaninterviewdoneinpersonoronthetelephone.Allsurveysconsistofquestionsthatincludetheopportunitytorespond.Thatiswhythetermquestionnaireisoftenusedinterchangeablywiththetermsurvey.Togetaccuratesurveydata,youmusttakeintoaccountthesurvey’squestions,responsechoices,samplingmethods,responserate,design,anddataanalysis.Surveyresultsarepresentedaswritten(printedonpaperorreproducedelectronically)andoralreports.Areliablesurveyproducesconsistentinformation,whileavalidoneresultsinaccurateinformation.Mailsurveysareoftenusedbecausepeoplearefamiliarwiththem;however,theresponserateisoftendismalwithoutagreatdealoffollow-upandincentives.Interviewersneedtraining,whichmaybetime-consumingandcostly.TheInternetisanefficientmethodofreachingawideaudience,buttechnicalexpertiseisneededwhetheryoudoitonyourownoruseacommercialfirmtoassist.Onlinesurveys,likeallsurveys,requirecarefuladvanceplanning.Cellphonesurveyscanbecostly,anditisdifficulttosampleamongpeoplewhojusthavelandlines,thosewithlandlinesandcellphones,andthosewithonlycellphones.Surveysmaybedoneforspecificorgeneralpurposes.Surveyfindingsthatareneededformanypeopleandplaceswillrequirespecialattentiontohowtheyaredesigned.Allcompletewrittensurveysshouldbekeptinlockedfiles,andonlyalimitednumberofstaffshouldhaveaccesstothemonaneed-to-knowbasis.Attheconclusionofdataanalysis,thesurveysshouldbeshredded.Separateidentifyinginformation(e.g.,names,birthdates,SocialSecuritynumbers)fromsurveyresponsesbyassigningcodestoindividualsandusingthecodestolinkthemtotheirresponses.Onlinesurveytakersmaybegiventheoptionofchoosingtheirownidentificationnamesandpasswordswhenloggingintoasurvey.Ifemailaddressesareusedtoinformparticipantsaboutandprovidealinktothesurvey,theseshouldbemadeavailableonlytoaselectgroupofsurveyors.IfasurveyisforapublicorprivateagencythatisreceivingU.S.governmentfunds,therearestringentlegaldimensionsofinformedconsent,privacy,andconfidentiality.Thesedimensionsincludethefollowing:

Afairexplanationoftheprocedurestobefollowedandtheirpurposes.Adescriptionofanyrisksandbenefits.Anoffertoansweranyinquiries.Aninstructionthatthepersonisfreetowithdrawconsentandtodiscontinueparticipation

withoutprejudice.Informedconsenttoparticipateinasurveyimpliesawillingnesstocompleteitbecauserespondentsunderstandthesurvey’spurposeandprocedures,thepotentialrisksandbenefitsofresponding,whetherthereispaymentforparticipation,howconfidentialitywillbehandled,andwhetherwithdrawalwithoutpenaltyispossible.Asurvey’sprivacypolicyshoulddothefollowing:

Describethedatathatwillbestoredinthesurvey’sdatabase.Explainwhythedataarebeingstored.Explainwhethertheorganizationgives,sells,ortransfersinformationandifitdoes,towhomandunderwhichcircumstances.Informhowunauthorizedattemptstochangethesite’scontentsaremonitored.Discusswhomaintainsthesite,andifrelevant,explainhowcookiesareused.Aretheysessionspecific?Ifnot,canusersoptoutofthewebpagefeaturethatstoresthecookiesbeyondthesession?Haverulesforethicalresearchandsurveysthatareoftenstricterforchildrenthanforadults.

Theinformedconsentmodelwithchildrenrequiresparents(orlegalguardians)toprovidepermissionfortheirchildrentotakethesurveyandtoallowthesurveyortocontactthechildren.Childrenthenprovidetheirassenttobecomeparticipants.

Ifyouconductsurveysinternationally,culturallyappropriateproceduresmustbefollowedtoprotectsurveyparticipants.Ifyouarenotsurewhatisandwhatisnotethical,gototheAmericanAssociationforPublicOpinionResearchortheCouncilofAmericanSurveyResearchOrganizations.

ThinkAboutThisReadthedescriptionofeachsurveybelowandthenanswerthesequestions:

1. Whatisthesurveyabout?2. Namethesurveymethod.3. Isthisthemethodyouwouldhavechosentoachievethesurvey’sobjectives?Explain.4. Listthreeethicalconcernsthatmustbeaddressedineachofthesurveys.

Evaluatetheadequacyofthefollowingsectionofasurveyorganization’sprivacypolicy.Whichcriteriadidyouuseinmakingyourevaluation?

SurveyDescriptions

PredictingHeartProblemsfromtheStressoftheSeptember11,2001,TerroristAttacksontheUnitedStatesHealthresearcherswantedtoexaminethedegreetowhichstressresultingfromtheSeptember11,2001,terroristattacksontheUnitedStatesforecastedheartproblemsinthenation3yearsafter.Theyenlistedanationalsampleofadultsinaweb-basedsurveyoftheirstress.Thepeoplewhoparticipatedhadcompletedahealthsurveybeforetheattackssotheinvestigatorshad“baseline”informationonthem.Within2weeksoftheattacks,theparticipantscompletedwebsurveys,andtheycontinuedtodoso1,2,and3yearsafter.Theresearcherscontactedthesurveyparticipants’physicianstofindoutwhethertheyhaddiagnosedcardiovascular(heart)ailmentsoverthe3-yearperiod.TheresearchersfoundthatacutestressresponsestotheSeptember11

attackswereassociatedwitha53%increasedincidenceofcardiovascularailmentsoverthe3subsequentyears.

SchoolFurnitureandLower-BackPainThissurveywasdesignedtofindoutwhethersometypesofschoolfurniturepreventorcauselower-backpaininchildren.Fivehundredforty-sixschoolchildrenages14to17yearsansweredawrittenquestionnaireaboutsittingpositionsduringschoolhoursandthepresenceandseverityoflower-backpain.Thedimensionsandtheweightofthechildren’sschoolbagsweremeasured,aswerethetypesanddimensionsoftheschoolfurniture.Thefindingswerethatmorethanhalfoftheadolescentsexperiencedlower-backpainduringthepreceding3months,andaboutone-quarterreportedreduceddailyfunctionorcareseekingbecauseoflower-backpain.Lower-backpainoccurrencewasnotfoundtobeassociatedwiththetypesordimensionsoftheschoolfurnitureorbodydimensionsbutwaspositivelyassociatedwithcarryingtheschoolbagononeshoulder.

MaleandFemaleParticipationinJazzImprovisationThreehundredthirty-twosurveysweregiventostudentsenrolledinmiddleschool,juniorhighschool,highschool,college,andcommunityjazzprogramswithin60milesofamajormidwesternuniversity.Thesurveywasadministeredduringjazzbandrehearsalsandtookapproximately10minutestocomplete.Jazzbanddirectorsand/orstudentteachersadministeredthesurveytomiddleschoolandhighschoolparticipants,whowereinstructedtotakethesurveyhomeforaparent’ssignatureindicatingapprovalforthechild’sparticipationinthestudy.Thefindingsindicatedthatfemalesaresignificantlylessconfident,aremoreanxious,andhavelessself-efficacy(attitude)towardlearningjazzimprovisation.

SatisfactionWithPaidPersonal-AssistanceServicesTraditionalpublichomecareprogramsintheUnitedStatesrelyonpublicorprivateagenciestohireandfirehomecareworkers,scheduleanddirectservices,monitorqualityofcare,disciplineworkersifnecessary,andpayworkersandapplicablepayrolltaxes.Intheagency-directedmodel,clientscanexpresspreferencesforservicesorworkersbuthavenoformalcontroloverthem.Thissurveywasconcernedwithcomparingconsumer-directedversusagency-directedhomecareonsatisfactionwithpaidpersonalassistanceservicesamongMedicaidbeneficiariesinWashingtonstate.Thesurveywasprimarilyconductedthroughtelephoneinterviewsusingacomputer-assistedtelephoneinterviewingsystem,buttherealsowereafewin-personinterviews.Thesurveyfoundthat,amongtheolderpopulation,butnotyoungerpeoplewithdisabilities,beneficiariesreceivingconsumer-directedservicesweremoresatisfiedthanindividualsreceivingagency-directedcare.

TeenDrivers’LicensingPreferencesandPracticesThissurveyusedprobabilitysamplingmethodstoidentify1,38315-to18-year-oldsfromanationallyrepresentativesampleofU.S.households.Theteenscompletedanonlinesurveysothatthesurveyorscouldlearnaboutnationallicensingrates,interestinearlylicensure,andreasonsfordelay.Someoftheteenscamefromcellphone–onlyhouseholds;Internetaccesswasprovidedtothosewithoutit.Thesurveyfoundthatat16years,teenswereaboutequallydividedamongthoseinthelearnerstageandthosewitharestrictedorfulllicense.Forteensoldenoughtostartthelicensingprocess,lackofacar,costs,parentavailability,abilitytogetaroundwithoutacar,andbeingbusywithotheractivitieswereleadingreasonsfordelay.

Parents’ChoiceofTreatmentforTheirChildrenWithAutismParentsofchildrenwithautismspectrumdisordershavealimitednumberofevidence-basedtreatmentsfromwhichtochoose.Ninehundredandseventyparentsrespondedtoanonlinesurveythataskedaboutthetreatmentscurrentlyinuse,thosediscontinued,andreasonsfordiscontinuation.Thesurveyfoundthatmostfamiliesadoptmultipletreatmentapproaches.Parentsweremostlikelytodiscontinuenonevidence-basedtreatmentswhentheydidnotseeimprovementintheirchild’sfunctioning.

InformationSecurityTheSurveyOrganization’semployeesunderstandtheneedforuserprivacy,andwemaintainstrictsecurityprocedurestoprotectyourinformation.TheOrganizationhasappointedaPrivacyPolicyAdministratortomonitorprivacypractices.Accesstouserdataisstrictlylimitedtospecificindividualswhoaretrainedtorespectuserprivacy.Theaccessgiventotheseemployeesisrestricted

totheirneedofsuchinformationforbusinessreasons.Alogofthosewhoaccessedthedataismaintainedandmonitoredtopreventsecuritybreaches.

REFERENCESBowker,A.,D’Angelo,N.,Hicks,R.,&Wells,K.(2011).Treatmentsforautism:Parentalchoicesandperceptionsofchange.JournalofAutismandDevelopmentalDisorders,41(10),1373–1382.

Courtney,M.,Zinn,A.,Johnson,H.,&Malm,K.(2011).EvaluationoftheMassachusettsAdolescentOutreachProgramforYouthsinIntensiveFosterCare:Finalreport.

Holman,E.A.,Silver,R.C.,Poulin,M.,Andersen,J.,Gil-Rivas,V.,&McIntosh,D.N.(2008).Terrorism,acutestress,andcardiovascularhealth:A3-yearnationalstudyfollowingtheSeptember11thattacks.ArchivesofGeneralPsychiatry,65(1),73–80.

Skoffer,B.(2007).Lowbackpainin15-to16-year-oldchildreninrelationtoschoolfurnitureandcarryingoftheschoolbag.Spine,32(24),E713–E717.

Wehr-Flowers,E.(2006).Differencesbetweenmaleandfemalestudents’confidence,anxiety,andattitudetowardlearningjazzimprovisation.JournalofResearchinMusicEducation,54(4),337–349.

Wiener,J.M.,Anderson,W.L.,&Khatutsky,G.(2007).Areconsumer-directedhomecarebeneficiariessatisfied?EvidencefromWashingtonState.Gerontologist,47(6),763–774.

Williams,A.F.(2011).Teenagers’licensingdecisionsandtheirviewsoflicensingpolicies:Anationalsurvey.TrafficInjuryPrevention,12(4),312–319.doi:10.1080/15389588.2011.572100.

2THESURVEYFORMQuestions,Scales,andAppearance

OverviewTodecideonasurvey’scontent,youhavetodefinetheattitude,belief,value,orideabeingmeasured.Forexample,whatismeantbyfear?Byaliberalperspective?Byself-efficacy?Alsoaskthis:WhichquestionsmustbeaskedinavalidsurveyifIwanttomeasurefear(oraliberalperspectiveorself-efficacy)?CanIgettheinformationIneedfromasurvey,orissomeotherdatacollectionmethodabetterwaytogo?

Theresponsestoclosedquestionscantaketheformofyes-or-noanswers,checklists,andratingscales.Checklistsgiverespondentsmanychoices(e.g.,“checkallthatapply”).Ratingscalesrequirerespondentstorank(e.g.,“1=top,10=bottom”)ororder(e.g.,“1=definitelyagree,2=agree,3=disagree,4=definitelydisagree”)ideas.Respondentsmayalsobeaskedtogiveanumberforananswer(e.g.,“Whatisyourage?”).

Responsestoquestionsproducedatathatarecalledcategorical(e.g.,“yesorno”),ordinal(e.g.,“big,bigger,biggest”),andnumericalorcontinuous(e.g.,“20,21,22yearsold”).Thesurveyorhastoconsiderthetypeofdatathesurveyproducesinordertocomeupwithanappropriatemethodofanalyzingandinterpretingitsfindings.

Surveyorsareofteninterestedinresponsestoindividualitems,suchasthenumberofpeoplewhowillvoteforCandidateXorhowoftenwomenbetween65and80yearsofagevisitadoctorina3-monthperiod.Butsometimestheyareconcernedwithascoreonagroupofitemsthatcollectivelyrepresentrespondents’views,healthstatus,orfeelings.

Onlinesurveyshavemanyusefulfeaturessuchasdrop-downmenusandtheabilitytobuildontheanswertoonequestiontocreatethenext.Youcanalsoreachlargenumbersofpeoplewithoneclick.Considerthecosts,whichincludepurchasefeesandtimetolearntousenewsoftware.

TheContentIstheMessageOnceyouhavedecidedthatasurveyisthemethodyouwanttousetogatherdata,youmustconsiderthecontentortopicsitwillinclude.

Supposeyouareevaluatingayouthcenter’sprogressandyourmaintaskistofindoutwhethertheprogram’sobjectiveshavebeenachieved.Saythatoneoftheobjectivesistoraiseyoungpeople’sself-esteembyprovidingthemwitheducation,jobs,financialhelp,andmedicalandmentalhealthassistance.Supposealsothatyoudecidetosurveytheyoungprogramparticipantstofindoutabouttheirself-esteem.Howwouldyoudeterminewhichcontenttoinclude?

Toselectthecontentofasurvey,youhavetodefineyourtermsandclarifywhatyouneedandcangetfromaskingpeopleabouttheirviews.

DefinetheTermsManyhumanattitudesandfeelings,suchasself-esteem,aresubjecttoarangeofdefinitions.Doesself-esteemmeanfeelinggoodaboutoneselfand,ifso,whatdoesfeelinggoodmean?Thesurveyorcananswerquestionssuchasthisbyreviewingtheresearch(inthiscase,theliteratureonself-esteem)tofindoutwhatisknownandtheorizedaboutaconceptsuchasself-esteem,consultingwithexperts,ordefining

theconceptforhimselforherself.Theproblemwithusingyourowndefinitionisthatothersmaynotbeconvincedofitsvalidity.Whenusingatheoreticalconceptsuchasself-esteem,itisprobablybesttoadoptarespectedpointofviewand,ifpossible,analreadyexistingandtestedsurveyform.

Ofcourse,formanysurveys,youwillnotbemeasuringtheoreticalideas,butevensoyoumustdefineyourterms.Supposeyouareassessingacommunity’sneedsforhealthservices.Thetermsneedsandhealthserviceswouldcertainlyrequiredefinitionbecauseyoucandefinethemwithrespecttothetypeandnatureofservicesrequired(outpatientclinics?hospitals?homevisits?)andhowconvenient(timeofdaywhendoctorsshouldbeavailable)orhowcontinuoustheyshouldbe(mustthesamedoctoralwaysbeavailable?).

SelectYourInformationNeedsorHypothesesSupposetwosurveyorschoosethesamedefinitionofself-esteemfortheirevaluationstudyoftheyouthcenter.Dependingonthecircumstances,Surveyor1mightdecidetofocusonself-esteeminrelationtofeelingsofgeneralwell-being,whereasSurveyor2maybeconcernedwithfeelingsofself-esteemonlyastheymanifestthemselvesinschooloratwork.CertainlySurveyors1and2,withtheirdifferentorientations,willaskdifferentquestions.Theresultswillyielddifferentkindsofinformation.Surveyor1,withaconcernforgeneralself-esteem,maynotevencoverworkorschoolandwillnotbeabletoprovidedataonthesetopics.Surveyor2,withhisorherspecialinterestsinschoolandwork,probablywillnotprovideinformationonparticipantself-esteemwithrespecttopersonalrelationships.Themessagesrevealedbyeachsurveywillclearlybedifferent.

Sayyouareinterestedinwhetherparticipantsintheyouthcenterhadtheirgeneralself-esteemenhancedafter2yearsofparticipationintheprogramandthatyouhavedefinedyourtermstoconformtoanacceptedtheoryofadolescentpersonality.Tomakesureyougetallthedatayouneed,youmustaskthisquestion:WhatinformationdoIwantandmustthereforemakecertainIcollect?Remember,ifyoudonotaskforit,youcannotreportitlater!

Herearesometypicalquestionsthattheevaluatoroftheyouthcentercouldask:

Istherearelationshipbetweengeneralfeelingsofself-esteemandwhethertheparticipantisaboyorgirl?Doesparticipants’self-esteemdifferdependingonhowlongtheyhavebeenintheprogram?

Theevaluatorcanalsocomeupwithhypotheseslikethese:

1. Boysandgirlsdonotdifferintheirgeneralfeelingsofself-esteem.2. Participantswhohavebeenintheprogramlessthan12monthswilldifferintheirself-esteemfrom

thosewhohavebeenintheprogram12monthsormore.

Thesequestionsandhypothesessuggestthatthesurveymustgetdataonthreetopics:

1. Generalfeelingsofself-esteem2. Sexofparticipant3. Lengthofparticipationintheprogram

Ifanyofthesetopicsisomitted,thesurveyoroftheyouthcentercannotanswerthetwoevaluationquestions.

MakeSureYouCanGettheInformationInsomecases,peoplemaybereluctanttorevealtheiropinions.Theevaluatoroftheyouthcenter,forexample,maydiscoverthatmanyyoungpeoplearereluctanttoanswerquestionsabouttheirfeelings.Inothercases,potentialsurveyrespondentsmaysimplybeunabletoprovideanswers.Supposeyouwanttoaskparticipantswhohavebeenattendingtheyouthcenterforabout6monthsabouttheirattitudestowardschooljustbeforeenteringthecenter’sprogram.Manymayhaveforgotten.Aninabilitytoanswerquestionsisnotuniqueandisamajorproblemwithaccuratelypredictingvoters’preferencesinnationalelections.Itoftentakestimeforpeopletosettleonacandidate,andsomepeoplechangetheiropinionsseveraltimesoverthecourseofacampaign.Thatisonereasonthatpollsproduceresultsthatdifferamongthemselvesandfromonepointintimetoanother.

Ifyouarenotcertainyoucangettheinformationyouneedfromasurvey,removethetopicandfindanotherdatasource,suchasobservationsorreviewsofrecordssuchasdiariesandreportsofschoolattendanceorhospitaladmissions.

DoNotAskforInformationUnlessYouCanActonItInasurveyofacommunity’sneedsforhealthservices,itwouldbeunfairtohavepeopleratetheirpreferenceforatraumacenterifthecommunityisunabletosupportsuchaservice.Rememberthatthecontentofasurveycanaffectrespondents’viewsandexpectations.Whyraisehopesthatyoucannotorwillnotfulfill?

Onceyouhaveselectedthecontentandsetthesurvey’sboundaries,yournexttaskistoactuallywritethequestions.Writemorequestionsthanyouplantousebecauseseveralwillprobablyberejectedasunsuitable.Firstdraftsoftenhavequestionsforwhicheveryonegivesthesameanswerornoonegivesanyansweratall.Beforedecidingonthenumberandsequenceofquestions,youmustbesurethatyoucoverthecompletedomainofcontentyouhaveidentifiedasimportanttothesurvey.Youmaywanttokeepalistsuchasthefollowingoneusedbythesurveyorofparticipantsatisfactionwiththeyouthcenter.Asyoucansee,thissurveywillnotcoverstaffsensitivitybutwillfocusinsteadonconsideration,accessibility,andavailabilityofservices.

Example:PlanforSurveyofSatisfactionWiththeYouthCenter

WritingQuestions

Open-EndedandClosedQuestionsSurveyitemsmaytaketheformofquestions.

Example:Open-EndedQuestion1. Howcourteousarethepeoplewhomakeyourappointments?

____________________________________________________________________________

____________________________________________________________________________

____________________________________________________________________________

Ortheymaybewordedasstatements.

Example:ClosedQuestionCircleyouragreementordisagreementwiththefollowing:

2.Thepeoplewhomakemyappointmentsarecourteous.

Mark(X)onechoice:

Sometimessurveyitemsareopen-ended,meaningthattherespondentsagreetoanswerthequestionorrespondtothestatementintheirownwords.Question1isopen-ended.Atothertimes,surveyitemsforcetherespondenttochoosefrompreselectedalternativesasinQuestion2.

Theoverwhelmingmajorityofsurveysrelyonmultiple-choicequestionsbecausetheyareefficientandareoftenmorereliablethanotherquestiontypes.Theirefficiencycomesfrombeingeasytouseand

score.Also,theirreliabilityisenhancedbecauseoftheuniformdatatheyprovide;everyonerespondsintermsofthesameoptions(e.g.,“agreeordisagree”or“frequentlyorinfrequently”).

Open-endedquestionscanofferinsightintowhypeoplebelievethethingstheydo,butinterpretingthemcanbeextremelydifficult,unlesstheyareaccompaniedbyanelaborateclassificationsystemandpeoplearetrainedtocategorizethedatatheygetwithinthesystem.

Considerthesetwoanswerstoaquestionfromasurveyofparticipantsinanelementaryschoolteachingprogram.

Example:Open-EndedQuestionforElementarySchoolTeachingProgramQuestion:Whatwerethethreemostusefulpartsoftheprogram?

____________________________________________________________________________

____________________________________________________________________________

____________________________________________________________________________

Answers:

RespondentAInstructor’slecturesThefieldexperienceTextbook

RespondentBInstructorTeachingintheclassroom

Themostusefulpartwastheexcellentatmosphereforlearningprovidedbytheprogram.

ItisnoteasytocompareRespondentA’sandRespondentB’sresponses.RespondentBliststheinstructorasuseful.Doesthismeanthattheinstructorisausefulresourceingeneral,andhowdoesthiscomparewithRespondentA’sviewthattheinstructor’slectureswereuseful?Inotherwords,areRespondentsAandBgivingthesameanswer?RespondentAsaysthetextbookwasuseful.Ifonlyonetextwasusedintheprogram,thenRespondentsAandBgavethesameanswer.Butbecausethetworecordedresponsesaredifferent,someguessingorinterpretationofwhatismeantisnecessary.

RespondentsAandBeachmentionedsomethingthattheotherdidnot:fieldexperienceandlearningatmosphere.Ifthesewereequallyimportant,thentheycouldbeanalyzedindividually.Butsupposeneitherwasparticularlysignificantfromtheperspectiveofthesurvey’susers.Wouldtheythenbeassignedtoacategorylabeledsomethinglikemiscellaneous?Categoriescalledmiscellaneoususuallyareassignedallthedifficultresponsesandbeforeanalysisandinterpretationarequitecomplicated.Beforeyouknowit,miscellaneouscanbecomethelargestcategoryofall.

Althoughitmayberelativelyeasyforarespondenttoansweranopen-endedquestion,analysisandinterpretationarecomplicated.Thefollowingclosedquestioncouldhavebeenusedtoobtainthesameinformationwiththeaddedresultofmakingtheresponseseasytointerpret.

Example:ClosedQuestionforElementarySchoolTeachingProgram

1. Pleasetellusifyouaresatisfiedordissatisfiedwitheachofthesecomponentsofourprogram.

MakingtheDecision:Open-EndedVersusClosedQuestionsChooseopen-endedquestionswhenyouwanttogivetherespondentstheopportunitytoexpressopinionsintheirownwordsandyouhavetheinterestinandresourcestointerpretthefindings.Open-endedquestionstendtoberelativelyeasytocompose.

Chooseclosedquestionsfortheirrelativeeaseofscoring,analysis,andinterpretation.Closedquestionscanbedifficulttopreparebecausetheyrequirethatallrespondentsinterpretthemthesamewayandthatallrelevantchoicesareincluded,mutuallyexclusive,andsensible.

OrganizingResponsestoOpen-EndedSurveyItems:DoYouGetAnySatisfaction?Acommonuseofasurveyistofindoutwhetherpeoplearesatisfiedwithanewproduct,service,orprogram.Theiropinionsprovideimportantinsightsintowhynewideasorwaysofdoingthingsdoordonotgetused.

Oneopen-endedsetofquestionsthatisparticularlyappropriateforgettingatsatisfactionrequirescollectinginformationaboutwhatpeoplelikebest(LB)abouttheproductorserviceandwhattheylikeleast(LL).

HereishowtheLB/LLtechniqueworks:

Step1:AskingRespondents’OpinionsAskrespondentstolistwhatisgoodandwhatisbad.Alwayssetalimitonthenumberofresponses;forexample,“Listatleastonething,butnomorethanthreethings,youlikedbestabouttheconference.”Ifparticipantscannotcomeupwiththreeresponses,theycanleaveblankspacesorwrite“none.”Iftheygivemorethanthree,youcankeepordiscardtheextrasdependingontheinformationyouneed.

Insteadofaskingabouttheconferenceasawhole,youmaywanttofocusonsomeparticularaspect;for

example,“Listatleastonething,butnomorethanthreethings,youlikedbestabouttheworkshops.”

Step2:CodingLB/LLDataOnceyouhavealltheresponses,thenextstepistocategorizeandcodethem.Todothis,youcancreatecategoriesbasedonyourreviewoftheresponsesoryoucancreatecategoriesbasedonpastexperiencewithsimilarprograms.

Trytokeepthecategoriesaspreciseaspossible—thatis,morecategoriesratherthanfewer—becauseitiseasiertocombinethemlaterifnecessarythanitistobreakthemup.

Supposethesearetypicalanswersparticipantsgavetothequestiononwhattheylikedleastabouttheconference:

Somepeopledidallthetalking.Theinstructordidn’talwayslisten.Icouldn’tsayanythingwithoutbeinginterrupted.Therewastoomuchnoiseandconfusion.Someparticipantswereignored.Theinstructordidn’ttakecontrol.Ididn’tgetachancetosayanything.SmithandJonesweretheonlyoneswhotalked.Theinstructordidn’tseemtocare.Icouldn’thearmyselfthink.

Youmightcategorizeandcodetheseanswerasfollows:

ExampleLB/LL:ResponseCategories

Nowmatchyourcodesandtheresponses:

ExampleLB/LL:ParticipantResponses

Tomakesureyouassignedthecodescorrectly,youmaywanttoestablishtheirreliability.Aretheyclearenoughsothatatleasttworaterswouldassignthesamecodeforagivenresponse?

Step3:LB/LLDataWhenyouaresatisfiedaboutreliability,thenextstepistocountthenumberofresponsesforeachcode.

Hereishowtodothisstepfor10participants:

ExampleLB/LL:NumberofResponsesforEachCode

Lookatthenumberofresponsesineachcategory.The10participantslisted30thingstheylikedleastaboutthesmallgroupdiscussion.Seventeenof30(morethan50%)wereassignedtothesamecategory,

Code2,andthesurveyorcouldjustlyarguethat,basedonthedata,whattheparticipantstendedtolikeleastabouttheworkshopswasthatsomepeoplemonopolizedthediscussionsandothersdidnotgetachancetosayanything.

Next,countthenumberofparticipantswhoseanswerswereassignedtoeachcode.Forexample,onlyParticipantsAandBgaveanswersthatwerecoded1.

ExampleLB/LL:Participants’ResponsePattern

Lookatthenumberofparticipantswhoseresponsesfiteachcategory.Because8or9ofthe10participantsgaveresponsesthatfellintothesametwocategories(Codes2and3),theiropinionsprobablyrepresentthoseoftheentiregroup.Itissafetoaddthatparticipantsalsodislikedthedisorderlyatmospherethatprevailedduringtheworkshops.Theycomplainedthatthenoisemadeithardtothinkclearlyandtheinstructordidnottakecontrol.

Whenrespondentsagreewithoneanother,therewillbeonlyafewtypesofanswersandthesewillbelistedbymanypeople.Ifrespondentsdisagree,manydifferentkindsofanswerswillturnupontheirlistsandonlyafewpeople(fewerthan10%)willbeassociatedwitheachtype.

InterpretingLB/LLdatagetsmorecomplexwhenyouhavemanyparticipantsandresponsestocategorize.Suppose,forexample,youask100participantstoindicatewhichaspectsofahealtheducationprogramtheylikedbest.

First,youmustdecideonyourresponsecategoriesandassigneachoneacode.Thentrythis:

1. Putthecodesinrankorder.Thatis,ifthelargestnumberofparticipantschoseresponsesthatareassignedtoCode3,listCode3first.

2. Calculatethepercentageofstudentsassignedtoeachcode.If40of100studentsmaderesponsesthatwereassignedacodeof3,thenthecalculationwouldbe40%.

3. Countthenumberofresponsesassignedtoeachcode.4. Calculatethepercentageofresponsesassignedtoeachcode.If117responsesfrom400were

assignedtoCode3,then29.25%or117/400ofresponseswereforCode3.5. Calculatethecumulativepercentageofresponsesbyaddingthepercentagestogether:29.25%+

20.25%=49.50%.

Thefollowingisatablethatsummarizesthesestepswithsomehypotheticaldata.Thetableisthentranslatedintoalinegraphusingstandardsoftware.Thegraphisshownafterthetable.

ExampleLB/LL:SummaryofResponses

Asyoucansee,theresponsecategoriesarerankorderedalongthexaxisaccordingtothenumberofparticipantsassignedtoeachcode.Theyaxisrepresentspercentages.

Foreachresponsecategory,youshouldlookfortwopointsonthexaxis:thepercentageofparticipantsandthecumulativepercentageofresponses.First,thecumulativepercentagesofresponsesarejoinedwithasquare(□).Next,someofthepointsrepresentingpercentagesofparticipantsarejoinedwithadiamond().Thegraphshowsthatresponsescodedas3,4,and7seemtobecloseinresponsefrequency.Theyarethecategoriestobeemphasizedbecausethehighestpercentagesofparticipantswereassignedtothesecodesandtheyaccountfor69%ofallresponses.

Items8,10,and1formasecondintuitivegroupingthatrepresents23.5%ofallresponses.Takentogether,responsescodedas3,4,7,8,10,and1accountfor92.5%ofthetotal.

RulesforWritingClosedSurveyQuestionsMultiple-choice,closedsurveyquestionsconsistofastem,whichpresentsaproblem(typicallyintheformofastatement,question,briefcasehistory,orsituation),followedbyseveralalternativechoicesorsolutions.Herearerulesfortheirconstruction.

1.Eachquestionshouldbemeaningfultorespondents.Inasurveyofpoliticalviews,thequestions

shouldbeaboutthepoliticalprocess,parties,candidates,andsoon.Ifyouintroduceotherquestionsthathavenoreadilyobviouspurpose,suchasthoseaboutageorgender,youmightwanttoexplainwhytheyarebeingasked,forexample,“Weareaskingsomepersonalquestionssothatwecanlookforconnectionsbetweenpeople’sbackgroundsandtheirviews....”

2.Usestandardlanguagerules.Becauseyouwantanaccurateanswertoeachsurveyitem,youmustuseconventionalgrammar,spelling,andsyntax.Avoidspecializedwords(unlessyouaretestingpeople’sknowledgeofthem)andabbreviationsandmakesurethatyouritemsarenotsolengthythatyouareactuallytestingreadingorvocabulary.

Example:Item-WritingSkills—Length,Clarity,Abbreviations,andJargon

LengthPoor:Thepaucityofpsychometricscaleswithhighdegreesofstabilityandconstructvalidityismostbothersometosurveyorswhenmeasuringpeople’s

1. economiccharacteristics2. feelings3. knowledge4. health

Better:Thelackofreliableandvalidmethodscausessurveyorsthemostproblemswhenmeasuringpeople’s

1. economiccharacteristics2. feelings3. knowledge4. health

ClarityPoor:Whatremedydoyouusuallyuseforstomachaches?

Better:Whichbrandofmedicinedoyouusuallyuseforstomachaches?

AbbreviationsPoor:Whichpoliticalpartyisresponsiblefortheexpandingsizeofthegrossdomesticproduct(GDP)?

1. Republican2. Democratic

Better:WhichpoliticalpartyisresponsibleforthediminishingsizeoftheGDP?

1. Republican2. Democratic

JargonPoor:Inyourview,whichdyadismostresponsibleforfeelingsoftrustinearlychildhood?

1. Motherandfather2. Fatherandsibling3. Motherandsibling

Better:Inyourview,whichfamilycombinationismostresponsibleforfeelingsoftrustinearlychildhood?

1. Motherandfather2. Fatherandbrotherorsister3. Motherandbrotherorsister

3.Makequestionsconcrete.Questionsshouldbeclosetotherespondent’spersonalexperience.

Example:Item-WritingSkills—ConcreteQuestionsLessconcrete:Doyouthinkotherpeoplewouldenjoythebook?

Moreconcrete:Haveyourecommendedthebooktoanyoneelse?

Askingrespondentswhethertheythinkotherswouldenjoyabookismoreabstractthanaskingwhethertheyrecommendedittoothers.Whoaretheothers?Canyoubesuretherespondentknowsaboutotherpeople’senjoymentwithrespecttothebook?Thefurtheryouremoveaquestionfromtherespondent’sdirectexperience,thecloseryoucometotheproblemsassociatedwithremembering.

Considerthis:

Example:Item-WritingSkills—SpecificityofQuestionsFivecompaniesinasmallcityweresurveyedtofindoutabouttheirattitudestowardhiringwomenformanagerialpositions.Onesurveyquestionasked,“Doyouthinkwomenhaveasgoodachanceasmenformanagerialpositions?”Afriendofthesurveyorpointedoutthatabetterwayofaskingthequestionwas“At(fillinnameofcompany),dowomenhaveasgoodachanceasmenformanagerialpositions?”

Becarefulnottoerrontheconcreteside.IfyouaskpeoplehowmanyhoursofTVtheywatchedeachdayforthepastweek,youshouldbesurethatnoreasonexistsforbelievingthatthepastweekwasunusualsothatthedatawouldnotberepresentativeofatrueweek’sworthofTVviewing.Amongthefactorsthatmightaffectviewers’habitsareTVspecialssuchastheOlympicsandcataclysmssuchasplanecrashes,earthquakes,floods,andfires.

4.Avoidbiasedwordsandphrases.Certainnames,places,andviewsareemotionallycharged.Whenincludedinasurvey,theyunfairlyinfluencepeople’sresponses.Wordssuchaspresident,abortion,terrorist,andalcoholicareexamples.

Supposeyouaresurveyingpeoplewhohadjustbeenthroughadietprogram.Whichwordsshouldyouuse:thinorslender;portly,heavy,orfat?

Rememberthis?

Iamfirm.Youarestubborn.

Heisapig-headedfool.

Lookatthesequestions.

WouldyouvoteforRogerFields?WouldyouvoteforDr.RogerFields?WouldyouvoteforRogerFields,aliberal?

AlthoughRogerFieldsappearstobethemostneutraldescriptionofthecandidate,itmaybeconsideredtheleastinformative.YettheintroductionofDr.orliberalmaybiastheresponses.

5.Checkyourownbiases.Anadditionalsourceofbiasispresentwhensurveywritersareunawareoftheirownpositiontowardatopic.Lookatthis:

Example:Item-WritingSkills—HiddenBiasesPoor:Doyouthinktheliberalsandconservativeswillsoonreachagreaterdegreeofunderstanding?

Poor:Doyouthinktheliberalsandconservativeswillcontinuetheirpresentpoorlevelofunderstanding?

Whenyouareaskingquestionsthatyoususpectencouragestrongviewsoneitherside,itishelpfultohavethemreviewed.Askyourreviewerwhetherthewordingisunbiasedandacceptabletopersonsholdingcontraryopinions.Forasurveyofpeople’sviewsontherelationshipbetweentheliberalsandtheconservatives,youmightaskthefollowing:

Example:Item-WritingSkills—HiddenBiasesBetter:Inyouropinion,inthenext4years,howistherelationshipbetweentheliberalsandtheconservativeslikelytochange?

MuchimprovementSomeimprovementSomeworseningMuchworseningImpossibletopredict

6.Usecautionwhenaskingforpersonalinformation.Anothersourceofbiasmayresultfromquestionsthatmayintimidatetherespondent.Questionssuchas“Howmuchdoyouearneachyear?”“Areyousingleordivorced?”and“Howdoyoufeelaboutyourteacher,counselor,ordoctor?”arepersonalandmayoffendsomepeoplewhomightthenrefusetogivethetrueanswers.Whenpersonalinformationisessentialtothesurvey,youcanaskquestionsintheleastemotionallychargedwayifyouprovidecategoriesofresponses.

Example:Question-WritingSkills—VeryPersonalQuestionsPoor:Whatwasyourannualincomelastyear?

$____________

Better:Inwhichcategorydoesyourannualincomelastyearfitbest?

Below$10,000Between$10,001and$20,000Between$20,001and$40,000Between$40,001and$75,000Over$75,001

Categoriesofresponsesaregenerallypreferredforsensitivequestionsbecausetheydonotspecificallyidentifytherespondentandappearlesspersonal.

7.Eachquestionshouldhavejustonethought.Donotusequestionsinwhicharespondent’struthfulanswercouldbebothyesandnoatthesametimeorallowarespondenttoagreeanddisagreeatthesametime.

Example:Question-WritingSkills—OneThoughtperQuestionPoor:ShouldtheUnitedStatescutitsmilitaryordomesticspending?

YesNoDon’tknow

Better:ShouldtheUnitedStatessubstantiallyreduceitsmilitaryspending?

YesNoDon’tknow

or

ShouldtheUnitedStatesallocatemoremoneytodomesticprograms?

YesNoDon’tknow

or

IftheUnitedStatesreduceditsmilitaryspending,shoulditusethefundsfordomesticprograms?

YesNoDon’tknow

ResponsesforClosedQuestions

YesandNoTheresponsesinasurveywithclosedquestionscantakeseveralforms.

Example:Yes-and-NoResponsesHaveyougraduatedfromcollege?

YesNo

Doesyourcarhavefour-wheeldrive?

YesNoDon’tknow

Yes-and-noresponsesaresimpletouseandscore.Butaslightmisinterpretationmeansthattheanswerwillbeexactlyoppositefromwhattherespondentreallymeans.Also,insomecases,askingforabsolutelynegativeorpositiveviewsmayresultintheparticipant’srefusaltoanswerorchoiceof“don’tknow.”

RatingScales

CategoricalorContinuous?WhatAboutOrdinal?Aratingscalerequirestherespondenttoplacetheitembeingratedinanyoneofanorderedseriesofcategoriesoratsomepointalongacontinuum.Anumericalvalueisassignedtotheorderedcategoriesorpoints.Therearethreetypesofratingormeasurementscales.

1.Categorical.Thesearesometimescallednominalresponsescales.Theyrequirepeopletoaffirmornamethegroupstowhichtheybelong:gender,religiousaffiliation,school,orcollegelastattended.

Example:CategoricalRatingScaleWhatisthenewborn’sgender?

2.Ordinal.Thesescalesrequirethatrespondentsplaceanswersinorderofimportance.Aperson’seconomicstatus(high,medium,orlow)wouldprovideanordinalmeasurement.

Example:OrdinalRatingScaleWhatisthehighestlevelofeducationthatyouachieved?

3.Numericalscales.Numericalscalestaketwoforms:discreteandcontinuous.Adiscretescaleproducesaprecisenumber,whereasacontinuousscaleproducesanumberthatfallsonacontinuum.

Hereisadiscretescale:

Ondaysthatyoudrankalcoholduringthepast12months,howmanydrinksofalcohol(beer,wine,and/orhardliquor)didyouusuallydrink?

Theanswerisdiscretebecauseyougetanactualnumber.Contrastthistoanumericalscalethatproducesanumberonacontinuum:

Whatisyourbloodpressure?____/____mm/HgHowtallareyou(incentimeters)?________cm

Continuousdatacanhavealmostanynumericvalueandcanbemeaningfullysubdividedintofinerandfinerincrementsdependingontheprecisionofthemeasurementsystem.

Youcanalsogographictogetcontinuousdata.Graphicscalesareakindofratingscaleinwhichthecontinuumofresponsesisvisual.Perhapsthemostwell-knowngraphicscaleisusedindescribingpain(seeFigure2.1).Apersonisaskedtoplaceamarkonascaleortocitethenumberthatbestillustrateshisorcurrentpain.

Thedistinctionsamongcategorical,ordinal,andcontinuousscalesareimportantbecausetheydeterminethekindsofstatisticaltreatmentsyoucanuse.Consideraverages.Ifyouwanttheaveragescoreforcontinuousdata,youwouldusethearithmeticaverageormean.Forordinaldata,youmightconsidercalculatingthemedianorthenumberseparatingthehigherhalfofresponsesfromthelowerhalf(moreonstatisticalapproachesandsurveydatacanbefoundinChapter6).

Figure2.1PainMeasurementScale

Source:©1983Wong-BakerFACES®Foundation,www.WongBakerFACES.org.Usedwithpermission.OriginallypublishedinWhaley&Wong’sNursingCareofInfantsandChildren.©ElsevierInc.

Source:©1983Wong-BakerFACES®Foundation,www.WongBakerFACES.org.Usedwithpermission.OriginallypublishedinWhaley&Wong’sNursingCareofInfantsandChildren.©ElsevierInc.

OrdinalScalesWhenratersuseordinalscales,theyselectoneofalimitednumberofcategoriesthathavesomeordertothem:

Example:OrdinalScalesFrequentlySometimesAlmostneverVeryfavorable

FavorableNeitherfavorablenorunfavorableUnfavorableVeryunfavorableStronglyapproveApproveNeitherapprovenordisapproveDisapproveStronglydisapproveDefinitelyagreeProbablyagreeNeitheragreenordisagreeProbablydisagreeDefinitelydisagree

Ordinalscalesareeasytouseandinterpret.Howmanycategoriesshouldtherebe?Somepeopleuseasmanyas9categoriesandothersasfewas3(high,medium,andlow).Anevennumberofchoices—say,4—forcestherespondentawayfromthemiddleground(neitheragreenordisagree).Buttheneedsofthesurveyandskillsoftherespondentmustdeterminethenumberofcategories.Ifpreciseinformationisneeded,therespondentsarewillingandabletogiveit,andyouhavetheresourcestocollectit,usemanycategories(between7and9);otherwise,usefewer.

Considerthesetwosituations:

Example:SelectingtheNumberofCategories1.A4-minutetelephoneinterviewwasconductedtofindouthowoftenfamilieswithworkingmothersatedinnerinrestaurants.Thequestionwasasked:“Inatypicalmonth,howoftendoesyourfamilyeatdinnerinarestaurant?”Theresponsechoicesweretwoormoretimesaweek ,onceaweek ,orlessthanonceaweek .

2.Physicianswereaskedtoratetheappropriatenessofacomprehensivelistofreasonsforperformingselectedsurgicalproceduressuchascoronaryarterybypassgraftsurgeryandgallbladderremoval.Anappropriatereasonwasdefinedasoneforwhichbenefitstopatientsoutweighedrisks.Ascaleof1to9wasused,where1=definitelyinappropriate,whereas9=definitelyappropriate.

InSituation1,the4-minuteinterviewdictatedshortresponses.InSituation2,physicianswereaskedtousetheirexpertisetogivefairlyrefinedratings.

Anoften-usedordinalscaleistheLikert-typescaleinwhichrespondentsareaskedtotellhowcloselytheyagreeordisagreewithastatement.Thenumberatoneendofthescalerepresentstheleastagreement,orstronglydisagree,andthenumberattheotherendofthescalerepresentsthemostagreement,orstronglyagree.Likertscalestypicallyhave5points:

StronglydisagreeDisagreeNeitheragreenordisagreeAgreeStronglyagree

Sometimesa4-pointscaleisused;thisisaforced-choicemethodbecausethemiddleoptionofneitheragreenordisagreeor,byconvention,neutralisnotavailable.Dependingonthesurvey’sneeds,thelist

cangofromstronglydisagreetostronglyagreeortheotherwayaround.Ifthescaleincludesotherwordsateitherendtofurtherclarifythemeaningofthenumbers,itisknownasaLikert-stylequestion.

ResponsestoLikertitemsaresometimesanalyzedasordinaldata,particularlywhenthesurveyorcannotassumethatrespondentsperceivethedifferencebetweenadjacentratingpointsasequidistant.Whenregardedasordinaldata,Likertresponsescanbecollatedintobarchartsandsummarizedbythemedianorthemode(butnotthemean)andvariationsummarizedbytherange(butnotthestandarddeviation)oranalyzedusingnonparametrictests(e.g.,c2testorMann-Whitneytest).Ifthesurveyordecidestoregardthescaleascontinuous,thenmeansandstandarddeviationsareappropriatestatistics.Treatingthedataascontinuouscanoccurifthesurveyordecidestocompareaverageratings.DatafromLikertscalesaresometimesreducedtothecategoricallevelbycombiningallagreeanddisagreeresponsesintotwocategoriesofacceptandreject.Thec2andMcNemartestsarecommonstatisticalproceduresusedafterthistransformation(seeChapter6formoreonthesetechniques).

Forced-choicequestionsareoftenusefulwhenyouwanttodiverttherespondentfromtakingthepathofleastresistancebychoosingamiddlecategory(somewherebetweenagreeanddisagree—neutral)orclaiminguncertainty(notsure,don’tknow).Butforcingrespondentstochoosemayannoythemandmaynotuncoverthetruthabouttheirviews.

Someordinalscalesrelyonrelativejudgmentsorranks,asinthefollowingexample.

Example:RankOrderScaleSource:SurveyMonkey®

Thedownarrowstotheleftofeachnameconsistofthenumbers1to5.Therespondentcanselectaresponseonlyonce.IfFay

Grossisranked5,thentheothersmustberanked1,2,3,or4.

Rankinginvolvescomparingonefactortoanother.Anotherformthatanordinalscalecantakeisthecomparativescale.Thisoneaskstherespondenttocontrastasinglespecificobjectintermsofageneralsituation.

Example:ComparativeRatingScalePleasecomparetheImperialThaiRestauranttoothersinLosAngeles.

Checkone:

__Itisbetterthanmost.__Itisaboutthesameasmost.__Itisnotasgoodasmost.

Toensurethatrankordersandcomparativeratingscalesprovideaccurateinformation,youhavetobecertainthattherespondentsareinapositiontomakecomparisons.Dotheyhaveexperienceinjudgingwritingskills?Ifnot,howcantheyassignranks?AretheyfullyacquaintedwithrestaurantsinLosAngeles?Ifnot,howcantheymakecomparisons?

ChecklistAchecklistprovidesrespondentswithaseriesofanswers.Theymaychoosejustoneormoreanswersdependingontheinstructions.

Example:ChecklistResponsesinWhichRespondentMustChooseOneFromaListofSeveralWhichofthefollowingmedicinesdoyouprefermostfortreatingaheadache?

(Oneansweronly)

Example:ChecklistResponsesThatRespondentsAnswerYes,No,orDon’tKnowforEachIteminaListCheck(ü)oneboxoneachline.

Inthepast12months,hasadoctororotherhealthworkertoldyouthatyouhaveanyofthefollowing:

Checklistshelpremindrespondentsofsomethingstheymighthaveforgotten.Ifyousimplyaskedpeopletolisttheirmedications,chancesarethatsomewouldnotknow.Checklistsalsohelpwiththespellingofdifficultwords.Aproblemisthatrespondentsmightthinkachoicelooksfamiliarandcheckthewrongitem.Didtheytakepenicillinorampicillin?Wasitthismonthorlast?Also,itissomewhatdifficulttoformatandinterpretresponsestochecklistswheremultipleanswerscanbegiven.Supposeinthefirstexamplethatapersonchecksaspirinbutfailstoindicatewhethertheothermedicinesweretaken.Canyouassumethattheotherswerenottaken,orisitpossiblethattheywere,butthepersondidnotbothertocompletetheitem?

ChildrenandSurveysWhatdothesethreequestionshaveincommon?

1. Inthepast12months,howmanytimesonschoolgroundshaveyoubeenafraidofbeingbeatenup?Wouldyousay...

2. Haveyoudoneanyofthesethingsinthelastmonth?

3. Howhappyareyouwith...

Thesethreequestionswereallcreatedforchildrenorteens.Theyarebasedonquestionsthatareusedinrealsurveys.ThefirsttwocomefromsurveysledbytheU.S.OfficeofPlanning,ResearchandEvaluation,oftheAdministrationforChildrenandFamilies,andthethirdisbasedonaquestionfrom“Children’sWorlds,”aninternationalsurveyofchildren’swell-being.

Manygoodreasonsexistforsurveyingchildren.Asweallknow,childrenandadolescentsliveinadifferentworld,andweadultsmaynotreallyknoworunderstandwhattheythink,believe,orvalue.Surveyorsgenerallyagreethatifyouwanttolearnaboutchildren,youshouldgotothemdirectly.Also,manypeopleagreethatchildrenareactorsandparticipantsinsociety,notjustpassiveonlookers,soitmakessensetoincludetheminsurveys.Finally,surveyingchildrenandteenscanbeeducationalandfunforthemandforthesurveyor,too.

Herearesometipsforcreatingwrittenandonlinesurveysforchildrenandadolescents:

1. Keepquestionsshort.2. Withveryyoungchildrenbespecificandavoidabstractions.

Poor:Doyouusuallylikeyourschoollunch?

Comment:“Usually”isanabstractionanditmaybeadifficultconceptforyoungchildrentograsp.

Poor:Doyoulikethefoodatyourschool?

Comment:Achildmaylikethefoodatbreakfast,butdislikethefoodatlunch.Theword“food”istoovague.Specifyifyoumeanbreakfast,snacks,orlunch.Thequestionalsoneedstospecifyatimeperiod.

Better:Didyoulikeyourlunchatschooltoday?

or

Didyouliketheappleyouhadasasnacktoday?

3.Tryoutallquestionsbeforeyouusethem.Atrialwilltellyouifyouneedtoalterthesentencestructureorvocabularytomakeitmoreunderstandabletotheparticipants.Youmayfindthatyouneedtorevisethequestionsaltogether.Forexample,supposeatrialofthesurveyshowsthatfewerthanhalfofthechildreneatlunchatschool.Ifso,thequestion“Didyoulikeyourlunchatschooltoday?”willbeleftunansweredbyhalfthechildren.Ifyouwanttheinformation,youneedtorevisethesurveysothatyouprecedethequestionwith“Didyoueatlunchatschooltoday?”Ifyes,“Didyoulikeyourlunch?”

4.Askforresponsestopositivestatements.Itisdifficultconceptuallyforanyoneatanyagetodisagreewithanegativestatement.

Good:

Poor:

Supposethechildfeelssafeathome.Theresponse“false”translatesto:“ItisfalsethatIdonotfeelsafeathome.”Ifthechilddoesn’thaveagoodtimetogetherwiththefamily,theanswer“false”translatesto:“Itisfalsethatwedon’thaveagoodtimetogetherinmyfamily.”

5.Makecertainthatchildrenunderstandthattherearenowrongorrightanswers.Childrenandyoungpeoplemustbetoldwellinadvancewhytheyarebeingsurveyed(tohelpteachersknowhowyouliketolearnbest;tofindoutwhatkidsyouragedoafterschool)andhowtheinformationwillbeused(toseeifweneedtospendmoreclasstimeworkinginsmallgroups;tohelpusplanafterschoolactivities).

6.Donotforcechildrentoansweraquestioniftheydonotwantto.Thismeansthatyoumustensurethatinanonlinesurvey,ifachildwantstoskipQuestion1,heorshecanstillmoveontoQuestion2.Testingthesurveyinadvancewilltellyouiftherearequestionsthatoffendorstumpmostpotentialrespondents.

Ifso,thesequestionscanbeeliminatedfromthesurveyorhandledinanothermanner.Duringthesurvey’spreliminarytrial,youcanaskrespondentsiftheywouldanswerthequestionifitwereaskedindifferentformat(e.g.,inaprivateinterview).

7.Reinforceyourrespectfortherespondent’sprivacy.Tellthechildthatyouwillneverrevealindividualresponsestoanyonewithoutpermission.Allresponsesshouldbeanonymousor,failingthat,confidential.Olderchildrenmayunderstandthatallonlinedataareprobablytraceable.Theymayknowthatifyouusecommercialsurveysoftware,youarelikelytohavetheirIPaddresses,sotellthemhowyouplanonprotectingtheiridentity.Ifyouconductinterviews,makecertainthatyoudosoinaprivateplace.

OnlineSurveysOnlinesurveysarealmostalwaysself-administeredquestionnaires,asyoucanseefromtheexampleinFigure2.2.

Therulesforcreatingonlinesurveysandchoosingquestionsareexactlythesameasforotherself-administeredquestionnaires.Thesurveymustbedesignedtogettheinformationyouneedfromthepeoplewhoaremostlikelytogivethemostaccurateanswers.Thequestionsmustbegrammaticallycorrectwithareasonablerangeofoptionsforresponses.

Figure2.2Self-AdministeredQuestionnaire

Thedrop-downlistisaquestionformatthatisuniquetoonlinesurveys.Itisoftenusedtosavespacewhenalonglistofresponsesispossible.Forinstance,ifyouwanttoknowthecountryinwhichtherespondentlives,youcansaveagreatdealofscreenspacebyprovidingadrop-downlistofcountries,ratherthanprintingthelistitself.

Hereisaquestionaskingfortherespondent’sjobtitleinafictitiouscompanywithhundredsofemployees.Adrop-downlistisoftenusedtosaverespondentstimeandtoensurethestandardizationofresponses(seeFigure2.3).

Onlinesurveysoftencontainaprogressbar.Thebartellstherespondenttheproportionofthesurveytheyhavecompleted.Hereisanexample.

Example:SurveyProgressBar

Theprogressbargivesrespondentsinformationonhowmuchtimetheywillrealisticallyneedtocompletethesurvey.Ifrespondentsused10minutestocomplete20%ofthesurvey,thentheywillneed50minutestocomplete100%.

Someonlinesurveysareprogrammedtoallowrespondentstologoffandonatwill.Ifrespondentsneed50minutestocompletethesurvey,buttheyonlyhave30minutesatthistime,alog-on-and-offfeaturemayencouragethemtoreturntothesurveyatsomeothertime.GivingparticipantstheabilitytologonandoffatwillmaybeagoodideaifthereisapossibilitythattheirInternetconnectionmaybedisrupted.Bepreparedtosendemailreminderstorespondentswhologoffandthenforgetthattheyhavenotcompletedthesurvey.

Figure2.3OnlineDrop-DownList

Onlinesurveyquestionscanbetailor-madesothattheresponsetoonequestiondictatestheresponsetothenextasillustratedinthisexample.

Example:ResponsetoOneQuestionDictatestheResponsetotheNextSource:http://www.questionpro.com/help/162.html

http://www.questionpro.com/help/162.html

Onlinesurveysshouldexplainhowtogobacktochangeanswersandthenmoveforward.Considertheseinstructions:

Example:ExplanationforChangingOnlineResponsesPleasereadthefollowinginstructionsbeforebeginningthissurvey.

MovingForward:ClickontheNEXTbuttonlocatedatthebottomofthepagetosaveyourresponsesandcontinuetothenextpage.

MovingBack:UsethePREVIOUSbuttonlocatedatthebottomofthepagetoviewyourresponsesonapreviouspage.Youmaychangeyouranswerstopreviouslyenteredresponses.DonotusetheBACKbuttonofyourbrowsertoreturntothepreviouspage.Usingyourbrowser’sBACKbuttonmaycauseyoutoexitthesurvey.Shouldthisoccur,youwillneedtowait10minutestoreaccessthesurveyusingthelinkprovidedinyouremail.

PlainandSimpleSurveyQuestionsandResponsesSurveyquestionsandresponsechoicesshouldbewrittenata6th-to8th-gradereadinglevel(basedontheU.S.schoolgradesystem).Why?StudiesshowthattheaverageAmericanismostcomfortablereadingatthatlevel.Infact,theevidencesuggeststhatmostpeopleprefersimplewordsandsentencesinsurveys.

TherearemanyformulasavailabletohelpdeterminetheapproximatereadinglevelofadocumentandyoucanfindawealthofinformationabouteachofthemontheInternet.Mostreadabilityformulasprovideagrade-levelscoreandarebasedontheaveragenumberofsyllablesperwordandwordspersentence.Themoresyllablesthereareinawordandthemorewordsthereareinasentence,theharderitistoreadandunderstandthetext.

AmongthemostcommonlyusedmethodstodeterminereadabilityaretheFryandFlesch-Kincaidformulas.Eachhasitsadvantagesanddisadvantages,andexpertsdonotagreeonwhichisbetter.Youcangoonlinetolearnhowtouseeach.Usethesekeywordstogetstarted:“readabilitystatistics.”

TheFlesch-KincaidmethodisconsideredlessaccuratethantheFry,buttheFryistimeconsuming.YoucaneasilycalculatetheFlesch-ReadingeasescorebyusingthereadabilitytoolinMicrosoftWord.Herearetheinstructions.

1. Gotothe“Review”tab.2. Gotothe“Spelling&Grammar”tabandclick“Options.”3. Check“Showreadabilitystatistics”underthe“Grammar”heading.

Eachtimeyoucheckspellingandgrammar,youwillbegivenseveralreadabilitystatistics,includingtheFlesch-Kincaidgrade-levelscore.

Scaling

AdditiveScalesMostsurveysaredesignedsothateachindividualitemcounts.Inasurveyofpeople’sattitudestowardlivinginatrailerpark,youmightask12questions,eachofwhichisdesignedtobeusedtoanalyzeattitudesandthereforeisscoredseparately.Supposeyoucollectedinformationlikethis:

LengthoftimelivinginthistrailerparkWhetherpersoneverlivedinatrailerparkbeforeSatisfactionwithpark’saccom-modationsSatisfactionwithqualityoflifestyleAmenitiesintrailerAgeoftrailerAgeofcarTypeofcarAgeofrespondentGenderofrespondentAnnualincome

Withthisinformation,youcouldreportoneachfactindividuallyoryoucouldlookforrelationships:

BetweenageofrespondentandsatisfactionwithqualityoflifestyleBetweengenderandlengthoftimelivinginthetrailerpark

Othersurveysaredifferent,however,inthattheitemsdonotcountindividually;theymustbecombinedtogetascore.

Considerthis:

Example:ASurveyWithanAdditiveScaleDoctorsatUniversityMedicalCenterobservedthatmanyoftheirveryillpatientsappearedtofunctionquitewellinsociety.Despitetheirdisabilities,theyhadfriendsandwenttothemovies,shopping,andsoon.Otherpatientswithsimilarproblemsremainedathome,isolatedfromfriendsandfamily.Thedoctorshypothesizedthatthedifferencebetweenthetwogroupsofpatientswasintheirpsychologicalfunctioningandaccesstotheresourcesthatmakelifealittleeasierforeveryone.Aspartoftestingtheirhypothesis,theyplantogivethetwogroupstheFunctionalStatusInventorydevelopedjointlybytheHerbertMedicalSchoolandCaliforniaUniversity.After5yearsofstudyandvalidation,researchersatthetwouniversitieshavepreparedasurveyonfunctioningforusewithchronicallyillpatients.Highscoresmeangoodfunctioning;lowscoresmeanpoorfunctioning.

Themethodsusedtoproduceanadditivescalerequiresophisticatedsurveyconstructionskillsbecauseyouhavetoproveconclusivelythathighscorersareinactualitydifferentfromlowscorerswithrespecttoeachandeveryitem.

DefiningAdditiveScalesSurveyorsusethetermscaleinatleasttwoways.Thefirstreferstothewaytheresponsesareorganized:

1. Doyoueatsixormoreservingsoffruitsorvegetableseachday?

Theresponsescaleisthefollowing:Yes □No □

2. Howsatisfiedareyouwiththeexamplesinthisbook?

Theresponsescaleisthefollowing:

Theseconduseofthetermscalereferstoonequestionoracollectionofquestionswhosescoresaremeaningful.Youhaveaquality-of-lifescalewhenasurveywith10questionscanbescoredsothatascoreof1meanslowqualityandascoreof10meanshighquality.

Example:ASurveyofForeignLanguageSkills

Circlethecategorythatbestdescribesyourabilitytospeakeachofthefollowinglanguages.

Foreachitem—French,German,andsoon—anordinalratingscaleisusedtoorganizeresponses.Atthesametime,byaddingallfiveitemstogether,ascaleoflanguageabilitycanbederived.Arespondentwhoisfluentinalllanguageswouldbeatoneendofthescaleandonewhoisnotfluentinanylanguageswouldbeattheother.Supposeyouassignedtwopointsforeachlanguagemarkedfluent,onepointeachforthosemarkedsomewhatfluent,andnopointsfornotfluent.Apersonfluentinallfivelanguagescouldgetamaximumscoreof10points,whereassomeonewhowasfluentinnonewouldbeassignedtheminimumscoreof0.Inthisbook,thistypeofscaleiscalledadditive(becauseindividualresponsestoitemsareaddedtogether,thatis,combined).Amongthemostcommonlyusedadditivescalesarethedifferential,summated,andcumulative.

DifferentialScalesDifferentialscalesdistinguishamongpeopleintermsofwhethertheyagreeordisagreewiththeoryorresearch.Tocreateadifferentialscaleforanideasuchasequalityofopportunity,forexample,meansassemblingmanystatements(e.g.,“qualifiedmenandwomenshouldreceiveequalpayforequalwork”)andhavingexpertsrateeachstatementaccordingtowhetheritwasfavorabletotheidea.Nextyoucomputetheexperts’averageormedianratingsforeachstatement.Thenyouaskrespondentstoagreeordisagreewitheachstatement.Theirscoreisbasedonjustthoseitemswithwhichtherespondentagrees.Togettherespondent’sscore,youlookattheexperts’averagescoreforeachstatementchosenbytherespondent,addtheexperts’averages,andcomputethearithmeticmean.

Typically,thedirectionstousersofdifferentialscalesgosomethinglikethis:

Pleasecheckeachstatementwithwhichyouagree.

or

Pleasechecktheitemsthatareclosesttoyourposition.

Scoringadifferentialscalemighttakethisform.

Example:ScoringaDifferentialScaleStudentAwasadministeredthePhysicalFitnessInventoryandaskedtoselectthetwoorthreeitemswithwhichshemostcloselyagreed.Thesearethetwoitemsshechoseandthejudges’scores.

StudentA’sscorewas3.6(theaverageof3.2and4.0),whichwasconsideredtobesupportiveofphysicalfitness.(Thebestpossiblescorewas1.0andtheworstwas11.0.)

Aretheredisadvantagestodifferentialscales?Perhapsthemostobviousoneisintheamountofworkneededtoconstructthem.Also,youmusttakeintoaccounttheattitudesandvaluesofthejudgeswhoseratingsareusedtoanchorthescaleandinterprettheresponses.Thejudgesmaybequitedifferentfromthepeoplewhomightusethescale.

SummatedScalesAsummatedscalealignspeopleaccordingtohowtheirresponsesaddup.Supposeaself-esteemquestionnairehasaseriesofitemsthatusethesameratingscale(agree,neutral,disagree).

Example:CreatingaSummatedScaleforaSelf-EsteemSurvey

Directions:Checkifyouagreeordisagreewitheachofthefollowingstatements.

Howwouldyoucomputeasummativescaleforthisquestionnaire?First,decidewhichitemsarefavorable(inthiscase,b,d,ande)andwhicharenot(aandc).Next,assignanumericalweighttoeachresponsecategory.Youmightdosomethinglikethis:

favorable=+1pointinbetweenorneutral=0pointsunfavorable=–1point

Aperson’sscorewouldbethealgebraicsumofhisorherresponsestofiveitems.TheanswersthatPersonXgaveareshownintheexamplethatfollows.

Example:ScoringaSummatedScale

PersonXdisagreedwithItemA,whichisfundamentallyunfavorable,andgotascoreof+1.ForItemB,thepersonwasneutralandsoearnedascoreof0.ItemCproducedagreement,butitwasfundamentallyunfavorable;PersonXgotascoreof–1.TherewasagreementwithItemD,resultinginascoreof+1andaneutralresponsetoItemE,producingascoreof0.PersonX’ssummatedscalescorewas+1outofapossibletotalof+5.(Aperfectscoreof+5wouldhavecomeaboutifPersonXanswered:ItemA=disagree;ItemB=agree;ItemC=disagree;ItemsDandE=agree.)

SummingUpThecontentofasurveydependsoncleardefinitionsoftermsbecausehumanattitudesandfeelingsvaryacrosstheories.Becertainrespondentsareinapositiontoanswerthequestions.Aretheyinformedaboutthetopic?Donotaskforinformationthatyouwillnotusebecausethatisawasteofrespondents’andyourtime.Categoricalratingscalesaskforpeopletoaffirmtheirgroupaffiliations,includingsex,religiousaffiliation,school,orcollegelastattended.

Surveyquestionsmaybeclosedoropen-ended.Closedquestionswithseveralchoicesareeasiertoscorethanareopen-endedquestions.Open-endedquestionsgiverespondentsanopportunitytostateapositionintheirownwords;unfortunately,thesewordsmaybedifficulttointerpret.Whenwritingquestions,usestandardgrammarandsyntax.Keepquestionsconcreteandclosetotherespondents’experience;becomeawareofwords,names,andviewsthatmightautomaticallybiasyourresults;checkyourownbiases;donotgettoopersonal;anduseasinglethoughtineachquestion.Considerusingareadabilityformula.Numericalscalestaketwoforms:discreteandcontinuous.Adiscretescaleproducesaprecisenumber,whereasacontinuousscaleproducesanumberthatfallsonacontinuum.Ordinalscalesrequirethatrespondentsplaceanswersinorderofimportance.Anoften-usedordinalscaleistheLikertscaleinwhichrespondentsareaskedtotellhowcloselytheyagreeordisagreewithastatement.Onlinesurveysofferspecialquestion-and-responseformatssuchasdrop-downmenus.Itisimportanttomakesurethattherespondentknowshowtonavigatethroughanonlinesurvey,especiallyhowtomoveforwardandback(tochangeanswersorreview).Followtheseguidelineswhensurveyingchildren:keepquestionsshort;beconcreteandavoidabstractions;tryoutthesurvey;keepresponsechoicespositive;ensurethatchildrenunderstandtherearenorightorwronganswersandthattheydonothavetoanswerquestions;reinforceandproveyourrespectfortheirprivacy.

ThinkAboutThis1. Readthequestionsbelow.Tellwhethertheyusecategorical,ordinal,orcontinuousscales.

1. QuestionsaboutInternetuseforhealthinformation:

2. Prepareaself-administeredquestionnairewith10orfewerquestionsaskingstudentsabouttheir

satisfactionwiththecontentofthischapter.Useatleastonequestioneachthatwillyieldcategorical,ordinal,andnumericaldata.Justifythechoiceandclarityofeachquestion.

3. Createaself-administeredorinterviewquestionnaireforteensbetween13and17yearsthatasksthemaboutoneormoreofthesetopics:bullyingatschool,useofsocialmediaduringclasstime,andlifestylepractices(alcoholuse?Otherdrugs?Exercise?Diet?).

4. Explaintheseterms:open-endedquestion,closedquestion,forced-choicequestion,rankorderscale,comparativeratingscale,Likertscale,drop-downmenu,summativescale,anddifferentialscale.

3GETTINGITTOGETHERSomePracticalConcerns

OverviewHowlongshouldasurveybe?Asurvey’slengthdependsonwhatyouneedtoknow,whenyouneedtoknowit,theamountoftimerespondentsarewillingtospendansweringquestions,andyourresources.

Thefirstquestiononasurveyshouldbeclearlyconnectedtoitspurpose,objectivequestionscomebeforesubjectiveones,movefromthemostfamiliartotheleastandfollowthenaturalsequenceoftime,keepquestionsindependenttoavoidbias,putrelativelyeasyquestionsattheend(particularlyinlongsurveys)butput“sensitive”questionsinthemiddle,avoidmanyitemsthatlookalike,andplacequestionslogically.

Toboosttheresponseratesforself-administeredquestionnaires,planinadvanceandmonitorresults.Onlinesurveysystemscanletyoudothisinrealtime.Considersendingrespondentsaletterbeforethesurveybeginsandalsoatthetimeofthesurvey.Youcandothisviasnailmailoremail.Proposetosendsurveyresults,keepquestionnairesshort,andthinkaboutofferingincentives.Interviewsrequirepreparationandplanning.Interviewersneedsystematic,intensivetraining.Setupasystemformonitoringthequalityoftheinterviewsovertime.

Youshouldpilottestyoursurveytoseethatitcanbeaccessedandadministeredeasilyandaccordingtoplan.Yourmaingoalisreliableandvalidsurveydata.Reliabilityreferstotheconsistencyoftheinformationyouget,andvalidityreferstotheaccuracyoftheinformation.

Onewaytopromotethereliabilityandvalidityofyoursurveyistobaseitonanalready-validatedsurvey,butbecautious.Askforevidenceofthesurvey’sreliabilityandvalidity.

Apilottesthelpsyoudesignareliablesurvey.Whenpilottesting,anticipatetheactualcircumstancesinwhichthesurveywillbeconductedandmakeplanstohandlethem.Chooserespondentssimilartotheoneswhowilleventuallycompletethesurvey.Forreliability,focusontheclarityofthequestionsandthegeneralformatofthesurvey.

Inaworldthatencompassesmanylanguagesandcultures,surveyorsneedtolearnhowtotranslatesurveysbyrelyingonfluentspeakerstoassistandtotestthequestionsandresponsesformeaning.

Pilottestingbolstersreliabilityandvaliditybecauseitcanhelpyouseethatalltopicsareincludedandthatsufficientvarietyintheresponsesisavailable—ifpeopletrulydiffer,yoursurveywillpickupthosedifferences.

Mostpublicandprivateagenciesthatconductsurveysasstand-alonestudiesoraspartoflargerresearchactivitieshaverulesaboutprotectingtherespondents’privacyandconfidentiality.Bepreparedtoexplaintherisksandbenefitstorespondentsiftheycompletethesurvey.

LengthCountsThelengthofasurveydependsonwhatyouneedtoknowandhowmanyquestionsarenecessarysothattheresultinganswersarecredible.Anotherconsiderationistherespondents.Howmuchtimedotheyhaveavailable,andwilltheypayattentiontothesurvey?Relativelyyoungchildren,forexample,maystayputforonlyafewminutes.Youmustconsideryourresources.A10-minutetelephoneorface-to-faceinterviewwillcostlessthansimilarinterviewslasting20minutes.Onlinesurveyscanbegiventotensofthousandsofpeoplewithoneclick.Thetechnicalexpertiseneededtodesigninternetsurveysiscostly,however,andifyousignupwithacommercialvendor,youwillhavetopayalicensefeeandpossiblysetupspecialprivacyarrangements.Herearetwosituationsillustratinghowthecircumstancesunderwhichasurveyisconductedinfluenceitslength.

Inthefirstsituation,thelibraryisdevotingconsiderabletimetoconductinterviewstoidentifycommunity

needs.Inthesecond,thesurveyisshortenedconsiderably.

Example:HowaSurvey’sCircumstancesCanInfluenceItsLengthSituation1:Thelocallibrarywantstobesurethatitcontinuestomeettheneedsofachangingcommunity.Inrecentyears,manymoreofitspatronsaremorethan65yearsofageandasubstantialpercentagespeaksEnglishasasecondlanguage.Amongthelibrary’sconcernsaretheadequacyandrelevanceoftheirexhibits,newspapers,magazines,andotherperiodicals;programsfeaturingnewbooksandwriters;andspecial-interestcollectionsconcernedwithissuessuchashealth.Abilingualvolunteerwillbedevotingtwomorningsandoneafternoonaweekfor8weekstoa45-minuteface-to-faceinterviewwithusersofthelibrary.A50-itemsurveyformhasbeendesignedforthepurpose.

Situation2:Theneighborhoodlibraryalsowantstobesurethatitsservicesareappropriateforapopulationthatisincreasinglyolderandnon-Englishspeaking.Thecityhasdecidedthattheneighborhoodlibraryisnottheonlylibraryinthecitythatischanginginitsneedsandhasagreedtosponsorasurveyofitslibrarypatrons.Tominimizetheamountoftimethatlibrariansandpatronswillhavetospendonthesurvey,a10-minute,6-item,self-administeredquestionnaireispreparedbythecentrallibraryoffice.Fourquestionsareaskedabouttheadequacyofthelibrary’scollectionofbooksandmagazines,computeraccess,andspecial-interestcollections;andtwoquestionsaskabouttherespondent’seducationalbackgroundandincome.Tofacilitatethesurvey’sefficiency,questionnairesareobtainableatthecheckoutdesk,completedatthelibrary,andleftwiththelocalbranchlibrarianwhothensendsthemtothecentralofficeforanalysis.

GettingtheSurveyinOrderAllsurveysshouldbeprecededbyanintroductionandthefirstsetofquestionsshouldberelatedtothetopicdescribedinit.Lookatthisintroductionandfirstquestionforatelephoneinterview.

Example:AnIntroductiontoaTelephoneSurveyandItsFirstQuestionHello.IamcallingfromtheCaliforniaUniversity.Wearesurveyingpeoplewholiveinstudenthousingtofindoutwhetheritisasatisfactoryplacetolive.Yournamewasselectedatrandomfromthehousingregistry,adirectoryofstudentswhohavevoluntarilylistedtheirtelephonenumbers.Ourquestionnairewilltakenomorethan4minutes.Youcaninterruptmeatanytime.MayIaskyouthequestions?

[IfYES,continue.IfNO,saythankyou,expressconcernforbotheringtheperson,andhangup.][Continuehere.]

Thefirstquestionasksyouaboutyouroverallsatisfactionwithyourapartment.Doyouconsiderit[readchoices]

DefinitelysatisfactoryProbablysatisfactoryProbablynotsatisfactoryDefinitelynotsatisfactory[DONOTSAY]noopinionordon’tknow/wronganswer

Theinterviewerstartsoffbysayingthatquestionswillbeaskedaboutsatisfactionwithstudents’housing,andthefirstquestioncallsforaratingofsatisfaction.Peoplesometimesrespondbestwhenthefirstquestionsaskforobjectivefacts.Oncetheybecomeusedtothesurveyandmorecertainofitspurposes,theyaremorelikelytoprovidetheanswerstorelativelysubjectivequestions.Supposeyouwanttoknowaboutthesuccessofasummercitycleanupprogram,forexample.Youmightfirstbeginbyaskingparticipantshowtheyfirstheardabouttheprogramandhowlongtheyhadbeeninit(twoquestionsoffact)andthenaskhowwelltheylikedtheirjob.

Questionsshouldproceedfromthemosttotheleastfamiliar.Inasurveyofneedsforhealthservices,itemscanfirstbeaskedabouttherespondent’sownneedsforservices,thenthecommunity’s,thestate’s,andsoon.

Questionsofrecallshouldalsobeorganizedaccordingtotheirnaturalsequence.Donotaskgeneralquestions,suchas“Whendidyoufirstbecomeinterestedinjogging?”or“Whydidyouchoosejoggingoverotherphysicalexercise?”Instead,prompttherespondentandask,“Whenyouwereinhighschool,didyouhaveanyinterestinjogging?Incollege?”

Sometimestheanswertoonequestionwillaffectthecontentofanother.Whenthishappens,thevalueofthequestionnaireisseriouslydiminished.Lookatthis:

Example:OrderingSurveyQuestions(1)Whichquestionshouldcomefirst?

1. Howmuchhelpdoesyouradvisergiveyou?2. Whichimprovementsdoyouwantinyoureducation?

Answer:QuestionBshouldcomebeforeQuestionA.Ifitdoesnot,thenadviser–studentrelationsmightbeemphasizedundulysimplybecausetheyhadbeenmentioned.

Here’sanotherexample.Whichquestionshouldcomefirst?

Example:OrderingSurveyQuestions(2)1. Howsatisfiedareyouwiththepresident’seconomicpolicy?2. Whatisthequalityofthepresident’sleadership?

Answer:QuestionBshouldprecedeQuestionAbecauseapersonwhoisdissatisfiedwiththepresident’seconomicpolicy(andperhapsnothingelse)mightratethequalityofthepresident’sleadershiplowerthanotherwise.

Placerelativelyeasy-to-answerquestionsattheendofasurvey.Whenquestionnairesarelongordifficult,respondentsmaygettiredandanswerthelastquestionscarelesslyornotanswerthematall.Youcanplacedemographicquestions(age,income,gender,andotherbackgroundcharacteristics)attheconclusionbecausethesecanbeansweredquickly.

Avoidmanyitemsthatlookalike.Twentyitems,allofwhichasktherespondenttoagreeordisagreewithstatements,mayleadtofatigueorboredomandtherespondentmaygiveup.Tominimizelossofinterest,groupquestionsandprovidetransitionsthatdescribetheformatortopic,asinthefollowingexample.

Example:ProvidingTransitionsThenext10questionswillaskwhetheryouagreeordisagreewithdifferentplanksoftheDemocraticPartyplatform.

Questionsthatarerelativelysensitiveshouldbeplacedtowardtheend.Topicssuchasgroominghabits,religiousviews,andpositionsoncontroversialsubjectssuchasabortionandguncontrolmustbeplacedfarenoughalongsothereisreasontobelievetherespondentiswillingtopayattentionbutnotsofarthatheorsheistoofatiguedtoanswerproperly.

Finally,questionsshouldappearinlogicalorder.Donotswitchfromonetopictoanotherunlessyouprovideatransitionalstatementtohelptherespondentmakesenseoftheorder.

Hereisachecklistofpointstoconsiderinselectingtheorderforthequestionsinyoursurvey:

ChecklisttoGuideQuestionOrderForanygiventopic,askrelativelyobjectivequestionsbeforethesubjectiveones.Movefromthemostfamiliartotheleast.

Followthenaturalsequenceoftime.Seetoitthatallquestionsareindependent.Relativelyeasy-to-answerquestions(includingdemographicquestionssuchasageoroccupation)shouldbeaskedattheend.Avoidmanyitemsthatlookalike.Sensitivequestionsshouldbeplacedwellafterthestartofthesurveybutalsowellbeforeitsconclusion.Questionsshouldbeinlogicalorder.

QuestionnaireFormat:AestheticsandOtherConcernsAquestionnaire’sappearanceisvitallyimportant.Awrittenself-administeredquestionnairethatishardtoreadcanconfuseorirritaterespondents.Theresultislossofdata.Apoorlydesignedinterviewformwithinadequatespaceforrecordinganswerswillreducetheefficiencyofeventhemostskilledinterviewers.

Herearesomedosanddon’ts:

Do:Putjustonequestiononaline.Leaveplentyofspaceforresponses.Don’t:Squeezeseveralquestionstogether.Donotabbreviatequestions.

ResponseFormatThegeneralruleistoleaveenoughspacetomaketheappropriatemarks.Hereareseveralresponseformats.

Example:ResponseFormats

FormatBorCisprobablysafer.

BranchingQuestions,ortheInfamous“Skip”Pattern

Whathappenswhenyouareconcernedwithgettinganswerstoquestionsthatyouknowareappropriateforonlypartofyourgroup?Supposeyouaredoingasurveyofyoungpeople’sparticipationinafterschoolactivities.Youknowthatonemajoractivitymightbesportsandanothermightbemusic,butyoualsoknowonlysomeparticipateineither.

Ifyouwanttoaskaboutatopicthatyouknowinadvancewillnotberelevanttoeveryoneinthesurvey,youmightdesignaformsuchastheoneinthefollowingexample.

Example:SkipPatternsorBranchingQuestions3.Doyouparticipateinsports?

1. No(GOTOQUESTION4)2. Yes

IFYES,whichsportsdoyouperform?

or

Doyouparticipateinsports?Yes(CompleteSectionA)

or

No(GotoSectionB)

Youmustbeextracarefulinusingbranchingquestions(or,astheyareoftencalled,skippatterns)inwrittenself-administeredquestionnaires.Infact,somesurveyorsthinkthatskippatternsareconfusingtomostpeopleandshouldnotbeusedinwrittenquestionnairesatall.Remember,respondentscanskipimportantquestionsiftheyarebaffledaboutwheretogonextonthesurveyform.Interviewersmustbetrainedtofollowthebranchesorelsetheymightbeunabletoadministerthesurveycorrectly.

Onlinesurveysareeffectivevehiclesforbranching(sometimescalled“piping”)becauseyoucandesignthesoftwaretoguidetherespondentandeverycommercialvendormakesiteasytouseskip-patternlogic.Forinstance,ifthesurveylogicis“IfrespondentanswersnotoQuestion3,thensendrespondenttoQuestion6,”therespondentwhoanswersnotoQuestion3willautomaticallyandpainlessly(fromtherespondent’sperspective)bemovedtoQuestion6.

Administration:WhoGivesWhattoWhom?

Self-AdministeredQuestionnairesSelf-administeredquestionnairesrequireagreatdealofpreparationandmonitoringtogetareasonableresponserate.Thesequestionnairesaregivendirectlytopeopleforcompletionandusuallylittleassistanceisavailableincasearespondentdoesnotunderstandaquestion.Asurveyquestionnaireaskingaboutteachers’availabilityandneedsforin-servicetrainingmaybeplacedinhisorherofficemailbox,forexample,withaweek’sreturnrequested.Ofcourse,teacherswhohavedifficultywiththeformmightdiscusstheproblemamongthemselves,butnoguaranteesexistthatthesolutionwillbejustascorrectorincorrectasiftheindividualactedalone.

Advancepreparation,intheformofcarefuleditingandtryouts,willunquestionablyhelpproduceaclear,readable,self-administeredquestionnaire.Tofurtherensurethatyougetwhatyouneed,youshouldreviewthereturns.Areyougettingtheresponserateyouexpected?Areallquestionsbeinganswered?Hereisachecklistforusingself-administeredquestionnairesthatcanhelpyougetwhatyouneed.

ChecklistforUsingSelf-AdministeredQuestionnairesSendrespondentsanadvanceletterviaregularmailoremailtellingthemthepurposeofyourquestionnaire.Thelettershouldnotifypeoplewhenthesurveyiscoming,explainwhytherespondentsshouldanswerthequestions,tellthemwhytheywereselected,howlongthesurveywilltake,andwhethertheywillbereimbursed.Prepareashort,formalexplanationtoaccompanythequestionnaireform.Ifyouhavealreadyinformedtherespondentaboutthesurveyinaletterorviaemail,youcankeepthisexplanationshort.Itshouldagaindescribethesurveyaims,respondents,andresponseburden.Considersendingrespondentsasummaryofthefindings.Ifyouaskquestionsthatmaybeconstruedaspersonal—suchasgender,race/ethnicity,age,orincome—explainwhythequestionsarenecessary.Keepthequestionnaireproceduressimple.Providestamped,self-addressedenvelopesformailsurveys.Keeppagefoldingtoaminimumsorespondentsdonotfeeltheyareinvolvedincomplicatedphysicalactivities.Keepquestionnairesshort.Askonlyquestionsyouaresureyouneedanddonotcrowdthemtogether.Inwrittensurveys,giverespondentsenoughroomtowriteandbesureeachquestionissetapartfromthenext.Considerincentives.Thismayencouragepeopletorespond.Incentivesmayrangefrommoneyandstampstopens,food,andfreeparking.Somesurveysenrollparticipantsinalotteryinwhichoneormore“winners”receiverelativelylargegiftslikecomputersandelectronicreaders.Bepreparedtofollowuporsendreminders.Theseshouldbebriefandtothepoint.Donotforgettobudgetmoneyandtimeformailingsandfollow-upphonecalls.Ifyouareusingonlinesurveysoftware,makesureyourcontractincludesafollow-upoption.Ifyouaredoinganonlinesurvey,herearesomespecialconsiderations.

Prepareallsurveyquestionsandresponsesbeforeyougoonlinetocreatethesurvey.Thesurveysoftwarewillguideyoutothetypeofquestion(suchasmultiplechoicewithoneanswerormultiplechoicewith“checkallthatapply”),butdonotletyourselfbeinfluencedbythesoftwareforthesakeofconvenience.Ifnecessary,provideinstructionsforcompletingeachquestion.Decideonwhethertokeepthesurveyasonecontinuousdocumentortodivideitintooneormorequestionsperscreen.Ifyouchoosetohaveoneormorequestionsperscreen,checkoneachscreentomakesurethatitisreadable.Lookateachquestiononline.Ifitisdifficulttoreadoruse,editit.Onceyouhaveallquestionsinadocument,reviewthedocumenttoensurethatthequestionsareproperlysequencedandeasytocomplete.Keeprespondentsuptodateontheirprogress.Onlinesurveysoftwareallowsyoutoprovideprogressgraphs.

Interviews

Findinginterviewers

Interviewersshouldfitinaswellaspossiblewithrespondents.Theyshouldavoidflamboyantclothes,haircuts,andsoon.Sometimesitisagoodideatoselectinterviewerswhoaresimilartorespondents.Ifyouwanttoaskadolescentgirlswhytheysmoke,forexample,youmighthireyoungwomentodothequestioning.

Itisalsoimportantthattheinterviewersbeabletospeakclearlyandunderstandably.Unusualspeechpatternsoraccentsmayprovokeunnecessarilyfavorableorunfavorablereactions.Theinterviewer’swayoftalkingisanextremelyimportantconsiderationinthetelephoneinterview.Youshouldbekeenlyawareofthepossibilitythattheinterviewer’sattitudetowardthesurveyandrespondentwillinfluencetheresults.Iftheinterviewerdoesnotexpectmuchandsendsthismessage,theresponseratewillprobablysuffer.Tomakesureyouaregettingthemostaccuratedatapossible,youshouldmonitortheinterviewers.

Traininginterviewers

Thekeytoagoodtelephoneorface-to-faceinterviewistraining,whichshouldensurethatallinterviewersknowwhatisexpectedofthemandthattheyaskallthequestionsinthesameway,withinthesameamountoftime.

Whetheryouaretrainingtwointerviewersor20,itisimportanttofindatimetomeettogether.Theadvantageofmeetingsisthateveryonecandevelopastandardvocabularyandshareproblems.Ifthetraineeshavetotraveltoreachyou,youmayhavetothinkaboutpayingforgasolineorothermeansofprivateorpublictransportation.

Onceatthetrainingsite,traineesmusthaveenoughspacetositandwriteorperformanyotheractivitiesyouwillrequireofthem.Ifyouwantthemtointerviewoneanotheraspracticefortheirrealtask,besuretheroomislargeenoughsothattwoormoregroupscanspeakwithoutdisturbingtheothers.Youmayevenneedseveralrooms.

Iftrainingtakesmorethananhourandahalf,youshouldprovidesomeformofrefreshment.Ifyoucannotaffordtodothat,atleastgivetraineestimetoobtaintheirown.

Traineesshouldbetakenstepbystepthroughtheirtasksandbegivenanopportunitytoaskquestions.Itisalsoessentialtotellthemsomeofthereasonsfortheirtaskssotheycananticipateproblemsandbepreparedtosolvethem.Themostefficientwaytomakesurethetraineeshavealltheinformationtheyneedtoperformtheirjobistoprepareamanual.Hereyoucanexplainwhattheyaretodoandwhen,where,why,andhowtheyaretodoit.Considersettingupagroupemailordesignatingasectionoftheproject’swebsiteasaplacetorecordchangesinsurveyproceduresandpolicies.

Conductinginterviews

Herearesometipsonconductinginterviewsthatshouldbeemphasizedinyourtrainingsessions:

1. Makeabriefintroductorystatementthatwilldescribewhoisconductingtheinterview(“MaryDoeforArmstrongMemorialMedicalCenter”),tellwhytheinterviewisbeingconducted(“tofindouthowsatisfiedyouarewithourhospitalityprogram”),explainwhytherespondentisbeingcalled(“We’reaskingarandomsampleofpeoplewhoweredischargedfromthehospitalinthelast2months”),andindicatewhetheranswerswillbekeptconfidential(“Yournamewillnotbeusedwithoutyourwrittenpermission”).

2. Trytoimpressonthepersonbeinginterviewedtheimportanceoftheinterviewandoftheanswers.Peoplearemorelikelytocooperateiftheyappreciatetheimportanceofthesubjectmatter.Donottrytodealwitheverycomplaintorcriticism,butsuggestthatallanswerswillreceiveequalattention.

3. Prepareyourselftobeflexible.Althoughitisimportanttostayonscheduleandaskallthequestions,afewpeoplemayhavetroublehearingandunderstandingsomeofthequestions.Ifthathappens,slowdownandrepeatthequestion.

4. Interviewpeoplealone.Thepresenceofanotherpersonmaybedistractingoraviolationofprivacyandthereforemayalterresults.

5. Ifusingaprintedinterviewsurvey,besuretoaskquestionsastheyappearintheinterviewschedule.Itisimportanttoaskeveryonethesamequestionsinthesamewayortheresultswillnotbecomparable.Ifusingalaptop,youwon’tneedtoworryabouttheorderofquestions.Youwillhavetolearnwhattodoifthecomputerfreezesorsomeothertechnicalglitchoccurs.

6. Interviewersshouldfollowallinstructionsgivenatthetrainingsessionanddescribedontheinterviewform.

7. Makecertaininterviewersunderstandtheinformedconsentprocess,andthattheyadheretoit.Someorganizationsandtheirethicsboardsrequiretheuseofspecificlanguage,whichtheinterviewermustusewithoutmodification.Forexample:

Ifyouhaveanyquestionsorconcernsregardingyourrightsasasubjectinthisstudy,youcancontacttheInstitutionalReviewBoard(IRB)at000-200-5138oraccesstheirwebsiteathttp://www.irb.wecare.org.

Monitoringinterviewquality

Tomakesureyouaregettingthemostaccuratedatapossible,youshouldmonitorthequalityoftheinterviews.Thismightmeansomethingasinformalashavinginterviewerscallyouonceaweekorsomethingasformalashavingthemsubmittoyouastandardizedchecklistofactivitiestheyperformeachday.Ifpossible,youmayactuallywanttogowithaninterviewer(ifitisaface-to-faceinterview)orspendtimewithtelephoneinterviewerstomakesurethatwhattheyaredoingisappropriateforthesurvey’spurposes.Topreventproblems,youmightwanttotakesomeorallofthefollowingsteps:

Establishahotline—someoneavailabletoansweranyquestionsthatmightoccurimmediately,evenatthetimeofaninterview.Providewrittenscriptsfortheinterviewer.Ifinterviewersaretointroducethemselvesorthesurvey,givethemascriptorsetoftopicstocover.Makesureyougiveoutextracopiesofallsupplementarymaterials.Ifdatacollectorsaretomailcompletedinterviewsbacktoyou,forexample,makesuretogivethemextraformsordisksandenvelopes.Asanoption,considerhavingdatacollectorsemailtheresultsorenterthemdirectlyontoasecurewebsite.Prepareaneasy-to-readhandoutdescribingthesurvey.Provideascheduleandcalendarsothatinterviewerscankeeptrackoftheirprogress.Numeroussoftwareprogramsareavailabletohelpyoudothis.Considerprovidinginterviewerswithvisualaids.Visualaidsmaybeextremelyimportantwheninterviewingpeoplewhoseabilitytospeakthelanguagemaynotbeasexpertasisnecessarytounderstandthesurvey.Visualaidsarealsousefulinclarifyingideasandmakingsurethateverybody

http://www.irb.wecare.org

isreactingtosimilarstimuli.Forexample,supposeyouwanttofindhowpeopleperceivetheeconomy:Isitimproving,remainingstable,orindecline?Toensurethateverybodyhasaccesstothesamesetofdataontheeconomy,youmightshowgraphsandtablesdescribingtheeconomytakenfromthelocalnewspaperforthelast1or2years.Anotheruseofavisualaidmightbeinasurveyofpeople’sidealexpectationsforaplannedcommunity.Youmightshowseveraldifferentplansandaskpeopletodescribetheirreactionstoeach.Thepreparationofaudiovisualaidsforuseinaninterviewisrelativelyexpensiveandrequiresthattheinterviewersbespeciallytrainedinusingthem.Considerthepossibilitythatsomeinterviewersmayneedtoberetrainedandmakeplanstodoso.

ReliabilityandValidity:TheQualityofYourSurveyArulerisconsideredtobeareliableinstrumentifityieldsthesameresultseverytimeitisusedtomeasurethesameobject,assumingthattheobjecthasnotchanged.Ayardstickthatshowsthatyouare6feet1inchtalltodayandalso2weeksfromtodayisreliable.

Peoplechange,ofcourse.Youmaybemoretired,angry,andtensetodaythanyouwereyesterday.Peoplealsochangebecauseoftheirexperiencesorbecausetheylearnedsomethingnew,butmeaningfulchangesarenotsubjecttorandomfluctuations.Areliablesurveyprovidesaconsistentmeasureofimportantcharacteristicsdespitebackgroundfluctuations.Itreflectsthe“true”score—onethatisfreefromrandomerrors.

Arulerisconsideredtobeavalidinstrumentifitprovidesanaccuratemeasure(freefromerror)ofaperson’sheight.Buteveniftherulersaysyouare6feet1inchtalltodayand6monthsfromnow(meaningitisreliable),itmaybeincorrect.Thiswouldoccuriftherulerwasnotcalibratedaccuratelyandyouarereally5feet6inchestall.

Ifyoudevelopasurveythatconsistsofnothingmorethanaskingahospitaladministratorhowmanybedsareinagivenwardandyougetthesameansweronatleasttwooccasions,youwouldhaveaninstrumentthatisreliable.Butifyouclaimthatthesamesurveymeasuresthequalityofmedicalcare,youhaveareliablesurveyofquestionablevalidity.Avalidsurveyisalwaysareliableone,butareliableoneisnotalwaysvalid.

IstheSurveyReliable?Inreviewingapublishedsurvey(alsoinassessingthequalityofahomemadeform),youshouldaskthefollowingquestionsaboutthreetypesofreliability:test-retest,equivalence,andinternalconsistency.

First,doesthesurveyhavetest-retestreliability?Onewaytoestimatereliabilityistoseewhethersomeonetakingthesurveyanswersaboutthesameonmorethanoneoccasion.Test-retestreliabilityisusuallycomputedbyadministeringasurveytothesamegroupontwodifferentoccasionsandthencorrelatingthescoresfromonetimetothenext.Asurveyisconsideredreliableifthecorrelationbetweenresultsishigh;thatis,peoplewhohavegood(orpoor)attitudesonthefirstoccasionalsohavegood(orpoor)attitudesonthesecondoccasion.

Second,arealternativeformsequivalent?Iftwodifferentformsofasurveyaresupposedtoappraisethe

sameattitude,youshouldmakesurethatpeoplewillscorethesameregardlessofwhichonetheytake.IfyouwanttouseFormAofthesurveyasapremeasure,forexample,andFormBasapostmeasure,checktheequivalenceofthetwoformstomakesureoneisnotdifferentfromtheother.

Equivalencereliabilitycanbecomputedbygivingtwoormoreformsofthesamesurveytothesamegroupofpeopleonthesamedayorbygivingdifferentformsofthesurveytotwoormoregroupsthathavebeenrandomlyselected.Youdetermineequivalencebycomparingthemeanscoreandstandarddeviationsofeachformofthesurveyandbycorrelatingthescoresoneachformwiththescoresontheother.Ifthevariousformshavealmostthesamemeansandstandarddeviationsandtheyarehighlycorrelated,thentheyhavehighequivalencereliability.Equivalencereliabilitycoefficientsshouldbehigh;lookforcoefficientsthatareasclosetoperfectaspossible.

Third,howinternallyconsistentarethesurveyquestionsinmeasuringthecharacteristics,attitudes,orqualitiesthattheyaresupposedtomeasure?Totestforinternalconsistency,youcalculateastatisticcalledcoefficientalpha(orCronbach’salpha,namedforthepersonwhofirstreportedthestatistic).Coefficientalphadescribeshowwelldifferentitemscomplementeachotherintheirmeasurementofthesamequalityordimension.

Manysurveyorsarenotatallconcernedwithinternalconsistencybecausetheyarenotgoingtobeusingseveralitemstomeasureoneattitudeorcharacteristic.Instead,theyareinterestedintheresponsestoeachitem.Decidewhetheryoursurveyneedstoconsiderinternalconsistency.Herearetwoexamples.

Whatisadequatereliability?Thecriteriondependsonthepurposeofthesurvey.Tocomparegroups(say,employeesatCompanyAwithemployeesatCompanyB),reliabilitycoefficientsof.50oraboveareacceptable.Tomakedecisionsaboutindividualeducationalorhealthneeds,youneedcoefficientsof.90.

Example:InternalConsistencyCountsA10-iteminterviewisconductedtofindoutpatients’satisfactionwithmedicalcareinhospitals.Highscoresmeanmuchsatisfaction,whereaslowscoresmeanlittlesatisfaction.Towhatextentdothe10itemsmeasurethesamedimensionofsatisfactionwithhospitalcare?

Example:InternalConsistencyDoesNotCountA10-iteminterviewisconductedwithpatientsaspartofastudytofindouthowhospitalscanimprove.Eightitemsaskaboutpotentialchangesindifferentservices,suchasthetypeoffoodthatmightbeserved;theavailabilityofdoctors,nurses,orotherhealthprofessionals;andsoon.Twoitemsaskpatientsfortheirage.Becausethissurveyisconcernedwithviewsonimprovingeightdifferentservicesandprovidingdataonageandeducationofrespondents,eachitemisindependentoftheothers.

IstheSurveyValid?Asurveyisvalidiftheinformationitprovidesisanaccuratereflectionofrespondents’knowledge,attitudes,values,andbehavior.Herearesomequestionstoaskaboutapublishedsurvey’svalidity:

Doesthesurveyhavepredictivevalidity?Youcanvalidateasurveybyprovingthatitpredictsanindividual’sabilitytoperformagiventaskorbehaveinacertainway.Forexample,amedical

schoolentranceexaminationhaspredictivevalidityifitaccuratelyforecastsperformanceinmedicalschool.Onewaytoestablishpredictivevalidityistoadministerthesurveytoallstudentswhowanttoentermedicalschoolandcomparethesescoreswiththeirperformanceinschool.Ifthetwosetsofscoresshowahighpositiveornegativecorrelation,thesurveyorinstrumenthaspredictivevalidity.Doesthesurveyhaveconcurrentvalidity?Youcanvalidateasurveybycomparingitagainstaknownandacceptedmeasure.Toestablishtheconcurrentvalidityofanewsurveyofattitudestowardmathematics,youcouldadministerthenewsurveyandanalreadyestablished,validatedsurveytothesamegroupandcomparethescoresfrombothinstruments.Youcanalsoadministerjustthenewsurveytotherespondentsandcomparetheirscoresonittoexperts’judgmentoftherespondents’attitudes.Ahighcorrelationbetweenthenewsurveyandthecriterionmeasure(theestablishedsurveyorexpertjudgment)meansconcurrentvalidity.Remember,aconcurrentvaliditystudyisvaluableonlyifthecriterionmeasureisvalid.Doesthesurveyhavecontentvalidity?Asurveycanbevalidatedbyprovingthatitsitemsorquestionsaccuratelyrepresentthecharacteristicsorattitudestheyareintendedtomeasure.Asurveyofpoliticalknowledgehascontentvalidity,forexample,ifitcontainsareasonablesampleoffacts,words,ideas,andtheoriescommonlyusedwhendiscussingorreadingaboutthepoliticalprocess.Contentvalidityisusuallyestablishedbyreferringtotheoriesaboutpersonality,emotions,andbehaviorandbyaskingexpertswhethertheitemsarerepresentativesamplesoftheattitudesandtraitsyouwanttosurvey.Doesthesurveyhaveconstructvalidity?Surveyscanbevalidatedbydemonstratingthattheymeasureaconstructsuchashostilityorsatisfaction.Constructvalidityisestablishedexperimentallybytryingthesurveyonpeoplewhomclinicalexpertssaydoanddonotexhibitthebehaviorassociatedwiththeconstruct.Ifthepeoplewhomtheexpertsjudgetohavehighdegreesofhostilityorsatisfactionalsoobtainhighscoresonsurveysdesignedtomeasurehostilityorsatisfaction,thenthesurveysareconsideredtohaveconstructvalidity.

SelectingandAdaptingSurveysTherearetensofthousands(maybemore)ofsurveysandsurveyquestionsthatyoucangetforfreeontheweb.Keepinmindthatthesehavebeendevelopedforgeneralaudiencesandmaynotbeappropriateforyoursurvey.Also,rememberthatyouhavenowayofknowingifthefreequestionshaveeverbeentriedoutoranalyzedand,ifso,withwhom.Perhapsmostimportantly,onceyouaccessasite’ssurveys,youmaybeaskedtosignaserviceagreementorbewarnedthat“cookies”willbeinstalledonyourbrowser.Thecostofthefreeitemsmaynotbeworthittoyou.

Hereisanexcerptfromarealserviceagreementthatmustbesignedinordertodownloadasurveytemplate.

Example:ExcerptFromaRealOnlineServiceAgreementforFreeSurveyItems(NamesChanged)Inordertooperateandprovidetheservice,wecollectcertaininformationaboutyou.Aspartoftheservice,wemayalsoautomaticallyuploadinformationaboutyourcomputer,youruseoftheservice,andserviceperformance.WeuseandprotectthatinformationasdescribedintheExtrasoftOnlinePrivacyStatement(http://go.extrasoft.com/fwlink/?LinkId=74170).Inparticular,wemayaccessordiscloseinformationaboutyou,includingthecontentofyourcommunications,inorderto(a)complywiththelaworrespondtolawfulrequestsorlegalprocess;(b)protecttherightsorpropertyofExtrasoftorourcustomers,includingthe

http://go.extrasoft.com/fwlink/?LinkId=74170

enforcementofouragreementsorpoliciesgoverningyouruseoftheservice;or(c)actonagoodfaithbeliefthatsuchaccessordisclosureisnecessarytoprotectthepersonalsafetyofExtrasoftemployees,customers,orthepublic.

TheserviceisaprivatecomputernetworkthatExtrasoftoperatesforthebenefitofitselfanditscustomers.Extrasoftretainstherighttoblockorotherwisepreventdeliveryofanytypeofemailorothercommunicationtoorfromtheserviceaspartofoureffortstoprotecttheservice,protectourcustomers,orstopyoufrombreachingthiscontract.Thetechnologyorothermeansweusemayhinderorbreakyouruseoftheservice.

GuidelinesforFindingUseableandUsefulSurveysintheResearchLiteratureOnewaytomakesurethatyouareusingareliableandvalidsurveyistorelyononethathasbeencarefullytestedbyothersurveyorsandresearchersinascientificstudy.Findingscientificallytestedmeasurescanbeparticularlyimportantifyouwanttosurveyattitudes,emotions,healthstatus,qualityoflife,andmoralvalues.Thesefactors,andotherslikethem,areelusiveanddifficulttomeasure,andproducingvalidmeasuresrequiresresearchexpertiseandfunding.Unlessyouarepreparedtodosuchastudy,youwillneedtofindasurveythatisreliableandvalidandappropriateforyourneeds.

Youcanfindsurveysinthepublishedliteratureandonlineonsitesdevotedtosurveysofpublicopinionorthosethatmaintaindataonthepublic’svalues,health,education,andwell-being(seeexampleonnextpage).

Manyresearchersinfieldslikehealth,education,andthesocialsciencesdevelop,validate,andusesurveysandpublishtheirfindingsinpeer-reviewedjournals.Thepublishedliteratureisanoutstandingsourceofready-madesurveysandsurveyquestions.Buthowdoyoufindthesesurveysburiedintheworld’svastandever-growingliterature?Andifyoufindthem,howdoyouselectamongthem?Answeringthesequestionsmeansknowinghowtosearchtheliteratureandevaluateitsqualityandappropriateness.Herearesevenguidelinestohelpyouthroughtheprocess:

1.Selectoneormoreonlinedigitallibrariesthatarerelevanttoyoursurvey.Onlinelibrariescontainmuchoftheworld’spublishedresearcharticles.CommonlyusedonlinelibrariesincludePubMed(medicineandhealth),ERIC(education),andtheWebofScience(socialscienceandscience).Thesedigitallibrariesprovidefreeaccesstocompletearticlesforsomejournals.Mostuniversitiesandsomebusinessesandpubliclibrariessubscribetooneormoreonlinereferencedatabases,andthroughthem,students,employees,andresearcherscanaccesstheworld’sjournals.ThePublicLibraryofScience(PLoS)consistsofopen-accessjournalsthatarefreetoeveryone.GoogleScholaralsoprovidesaccesstojournalarticlesthatcontainsurveys.GoogleScholarisnotspecificallydesignedforacademicandsurveypurposes,sofindingwhatyouneedmaytakelongerthanifyouuseadatabaseliketheWebofScience.

2.Specifythetopicsorvariablesyouwantyoursurveytocover.Comparethesetwoscenarios:

Surveyor1wantstofindoutwhystudentsinanewafterschoolprogramdropout.Surveyor1developstheAfterschoolDropoutSurvey.Thesurveyasksquestionsaboutfamilyincome,numberofchildreninthefamily,andeducationallevelofallfamilymembers.Surveyor1hypothesizesthatstudentswhodropoutaremorelikelytocomefromfamilieswithlowerincomesandlesseducationthanthosewhoremainintheprogram.

Surveyor2isalsointerestedinfindingoutwhythestudentsinthenewafterschoolprogramdropout.Surveyor2developstheAfterschoolDropoutSurvey.Thesurveyasksquestionsabouttheamountoftimestudentsspendeachdayreadingforpleasure,playingvideogames,orgoingonlinetocheckontheirfriends.Surveyor2hypothesizesthatthestudentswhospendthemosttimeplayingvideogamesorgoingonlinearemorelikelytodropoutthanarethosewhospendthemosttimereadingforpleasure.

Example:SearchforSchoolDropoutSurveys:SampleFindingsFromtheWebofScienceSource:ThomsonReutersWebofKnowledge.

Ifyouareinterestedinstudyingwhetherornotfamilyincomeandeducationareassociatedwithretentioninafterschoolprograms,thenSurvey1isagoodchoice.IfyouareinterestedintherelationshipbetweendroppingoutofanafterschoolprogramandhowmuchtimestudentsspendonactivitieslikevideogamesandFacebook,thenSurvey2istheoneforyou.NotethatbothSurveyor1andSurveyor2nametheirsurveythe“AfterschoolDropoutSurvey.”Surveyswiththesameorsimilartitlesmayactuallymeasurequitedifferentconcepts.Youneedtospecifyinadvancewhatyouwanttoknowandcarefullyevaluatetheappropriatenessofeachavailablesurvey.

3.Learnhowtosearchtheappropriateelectroniclibraries.Alldigitallibrariesprovidetutorialsforsearchingtheirdatabasesandarerelativelyuser-friendly.HereisanexampleofthesearchresultsforsurveysofschooldropoutsintheWebofScience.Youcanrefineyoursearchbychoosingthedisciplines,typeofarticle,languages,yearsofpublication,andsoon.

Onceyouhavethelistofpotentialarticles,checkeachonetobesureitcontainsasurvey.Ifyouneedto

continuethesearchbecauseyoudidnotfindappropriatesurveys,studythekeywordsinrelevantarticles.Youcanoftenfindthekeywordsnexttotheabstractorsummaryofthearticle.Eachdigitallibrarycataloguesitsarticlesusingsubjecttermsanddescriptors.Youcanlearnhowtousethesefromthelibrary’stutorials.

4.Makecertainthatyoucanadministerthesurveythewayitwasintendedorthatyouhavetheresourcestoadaptit.Supposeyoufindasurveyintheliteraturethatcoversthetopicsyouplanonaddressing.Youalsofindthatithasfaceandcontentvalidity.However,thesurveyisaface-to-faceinterviewandyouwantanonlinesurvey.Youneedconsiderableexpertisetotranslateaninterviewquestionnairetoanonlinesurvey.Oncedone,itmustbepilottestedextensively.Translation,pilottesting,andrevisionsbasedonthepilotcandelaythestartofyoursurveyandbecostly.So,eventhoughthesurveylooksgood,itmaynotbeappropriateforyourimmediateneeds.Youhavetolookfurther.

5.Checkonthecharacteristicsofthesurveyrespondents.Supposeyoufindanarticlewithasurveythatlooksreallygood.Asyoulookatthearticleinwhichthesurveywasused,askyourselfiftherespondentsaresimilartoyourplannedsample.Asurveythatisdesignedtofindouthowchildrencopewithviolence,forinstance,maybequitedifferentinlanguageandconceptfromonethatismeanttostudyadultcoping.

6.Selectthesurveythathasthebestavailableevidenceofreliabilityandvalidity.Thebestavailableevidencecomesfrommethodologicallyrigorousstudies.Thesearestudiesthatarecarefullycontrolled(say,randomizedcontrolledtrials),withjustifiableandmeaningfulsamples,appropriatestatistics,andsoundconclusions.Theydescribeandjustifytheirchoiceofallmeasures,includingsurveys.Itisoftenagoodideatogototheoriginalsourcelistedinthereferencestocheckonhowthesurveywasvalidated.

Thefollowinggivesyouthreesamplesofwrite-upsintheresearchliteratureregardingreliabilityandvalidity.

Example:SampleReportsofReliabilityandValidityintheResearchLiterature

1.Measuringloveinromanticrelationships:Ameta-analysisThisstudy(Graham,2011)aimedtoidentifythehigher-orderfactorsunderlyingthemostpopularmeasuresofloveinusetoday.Theresearchersusedamethodtheycalled“meta-analyticfactoranalysis.”Eighty-onestudiesrepresenting103samplesand19,387individualsprovideddataforthemeta-analysis.Thereportedcorrelationsbetweenseveralcommonlyusedmeasuresoflovewereaggregatedacrossstudiestoformameta-analyticcorrelationmatrix,whichwasthensubjectedtoprinciplecomponentsanalysis.Factorssuggestinggenerallove,romanticobsession,andpracticalfriendshipemerged.Lovewaspositivelyandobsessionwasnegativelyassociatedwithrelationshipsatisfactionandlength.Theresultscorroboraterecentconceptualizationsoflove(concurrentvalidity)andpointtowardtheneedformeasurementmethodsotherthanglobalself-reportinthefieldofloveresearch.

2.ReliabilityandConstructValidityofPsychopathyChecklistYouthVersionScoresamongIncarceratedAdolescentGirlsAlthoughrecentstudieshavedemonstratedtheutilityofassessingpsychopathictraitsinadolescentmales,thereissubstantiallylessevidenceontheutilityofassessingpsychopathictraitsinadolescentfemales.Thisstudy(Bauer,Whitman,&Kosson,2011)investigatedthereliabilityandconstructvalidityofthePsychopathyChecklist:YouthVersion(PCL:YV)amongasampleof80

incarceratedadolescentfemales.ReliabilityanalysesindicatedhighinterrateragreementandinternalconsistencyforPCL:YVscores.Consistentwithresearchonadultsandadolescentmaleoffenders,psychopathictraitswereassociatedwithagreaternumberofconductdisorderandalcoholdependencesymptoms,earlieronsetofcriminalbehavior,apropensitytowardinstitutionalviolence,andinterpersonalbehaviorsassociatedwithpsychopathy(constructvalidity).Consistentwithsomepriorstudiesofyouth,therewerenonegativerelationshipsbetweenPCL:YVscoresandinternalizingdisorderdiagnosesandsomepositiveassociationsbetweenpsychopathyratingsandnegativeaffectivity(constructvalidity).Resultssuggestthatassessingpsychopathictraitsinadolescentfemalesprovidesmeaningfulinformationregardingcriminalbehavior,patterns,andpersonalitytraitsinyouth.

3.Determinantsoffast-foodconsumption:AnapplicationofthetheoryofplannedbehaviorThisstudy(Dunn,Mohr,Wilson,&Wittert,2011)examinedthefactorsthatinfluencefast-foodconsumptioninAustralia.TheresearchersusedtheTheoryofPlannedBehavior(contentvalidity)tosurvey404peopleabouttheirattitudes,intentions,andbehaviors.Toexaminethefactorsthatpredictedfast-foodconsumption(predictivevalidity)theresearchersusedatechniquecalledstructuralequationmodeling.Theyusedothermethods,includingonecalledfactoranalysis,torefinethedefinitionsoftheconstructsthatcompriseattitudestowardfastfood.Theresearchersfoundthatfast-foodconsumptionmaybeinfluencedbyconvenience,satisfaction,familyandfriends,busylifestyles,experiencingcravingsforfastfood,notknowinghowtocook,workinglonghours,andeatingalone.People’sconcernforlong-termhealthisprobablyoverriddenbytheshort-termbenefitsoffastfood.

7.Tryoutthesurveywithasmallgroupofpeopletoseeifitisuseableanduseful.Thenpilottestitonalargergrouptobesure!

FindingSurveysontheWebSomeorganizationsoffersurveyquestionsfreeonlinetothepublic.Herearejustafewexamples.

GallupOrganization(http://www.gallup.com):

Onthissite,youcanfindrecentpoliticalandeconomicpollingquestionsandresults,aswellas“specialreports”thatexamineattitudestowardsocialissuesandconsumerbehavior.

GeneralSocialSurvey(http://www.norc.org/Research/Projects/Pages/general-social-survey.aspx):

TheGeneralSocialSurveyhasbeenconductedsince1972bytheNationalOpinionResearchCenter.Itmeasuresattitudestowardsocialandpublicpolicyissues,economicstatus,politicalevents,work,andfamilylife.

NationalOpinionResearchCenter(http://norc.uchicago.edu/homepage.htm):

NORCconductssurveyresearchonpublicpolicyissuesforavarietyofagenciesandcorporations.Seethe“Projects”sectionforcopiesofrecentsurveysandtheirreports.

ThePewCharitableTrusts(http://www.pewtrusts.org/):

Pewprovidesaccesstosurveysandsurveyquestionsconcerningthepubliconmediaandjournalism,religionandpubliclife,societyandtheInternetandglobalattitudes,andHispanicsinAmericaandpublicopinion.

http://www.gallup.com

http://www.norc.org/Research/Projects/Pages/general-social-survey.aspx

http://norc.uchicago.edu/homepage.htm

http://www.pewtrusts.org/

TheAgencyforHealthcareResearchandQuality(AHRQ)(http://www.ahrq.gov/data/dataresources.htm):

Atthesite,youcanfindinformationthatwillhelpyouwithsurveysonhealthcaredisparities,qualityandsafety,andHIVAIDS,tonameafewtopics.

TheU.S.Census(http://www.census.gov/):

Thisisamajorsourceofsurveyquestions.Regardlessofhowlargeorsmallasurveyis,andevenifittakesplaceoutsidetheUnitedStates,thisisanexcellentsitetoguidethedevelopmentandadaptationofsurveyquestionsaboutage,ethnicity,income,education,andoccupation.TheU.S.Censusalsoprovidessurveyquestionsthatarerelevanttocriminaljustice(http://www.census.gov/govs/cj/).Samplesurveysincludetheannualparoleandprobationsurveyandsurveysofsexualviolence.

TheInstituteofEducationSciencesthroughtheNationalCenterforEducationStatistics(http://nces.ed.gov/datatools/index.asp?DataToolSectionID=3):

Thissitehasanextensivelibraryofsurveys.Anexampleofthesurveysyouwillfindifyougotothesiteappearsonthenextpage.Asyoucansee,internationalsurveysareincluded.

Remember,buyer,beware!Becarefultochoosesurveysandindividualquestionsthatofferprooftheyhavebeenusedwithrespondentslikeyours.Someonlinesurveycompaniesprovidefreequestionsonarangeoftopicsfromcustomersatisfactiontohealthstatus.Manyofthesequestionshaveneverbeentestedorusedwithanyone,sotheirvalueisnotknown.Thebestsafeguardagainstpoorsurveysandsurveyquestionsistopilottestthemortrythemout.

Example:SearchforEducationSurveys:ASamplePageFromtheNationalCenterforEducationStatistics

http://www.ahrq.gov/data/dataresources.htm

http://www.census.gov/

http://www.census.gov/govs/cj/

http://nces.ed.gov/datatools/index.asp?DataToolSectionID=3

TheSurveyIsPutonTrial:GuidelinesforPilotTestingHerearesomebasicrulesforpilottestingasurvey:

1.Trytoanticipatetheactualcircumstancesinwhichthesurveywillbeconductedandmakeplanstohandlethem.Forinterviews,thismeansreproducingthetrainingmanualandallforms;formailedquestionnaires,youhavetoproduceanycoverletters,returnenvelopes,andsoon.Foronlinesurveys,youhavetotesttheeasewithwhichrespondentscanlogontoasecurewebsiteandaccessthesurvey.Needlesstosay,thisrequiresplanningandtimeandcanbecostly,somakesureyoursurveybudgetprovidesforthissetofactivities.

2.Youcanstartbytryingoutselectedportionsofthesurveyinaninformalfashion.Justthedirectionsonaself-administeredquestionnairemightbetestedfirst,forexample,orthewordingofseveralquestionsin

aninterviewmightbetested.Youmayalsowanttotryoutthesurveyprocessinitiallybyusingadifferentmethodfromtheoneyoueventuallyintendtouse.Soifyouareplanningtohandoutquestionnairestoconferenceparticipants,thetrialmayinvolveaninterviewsothatanyproblemswiththequestionsontheformcanbediscussed.Intheend,youshouldgivethesurveylogisticsandformafairpretrial.

3.Chooserespondentssimilartotheoneswhowilleventuallycompletethesurvey.Theyshouldbeapproximatelythesameage,withsimilareducation,andsoon.Donotincludepilottestrespondentsintheactualsurvey.Pilottestrespondentsmayfeelinvestedinthesurveybecausetheyhelpedtocreateit,and,becausetheyarefamiliarwiththesurvey’spurposeandlanguage,theymayhavepreconceivedideasabouthowtogivethe“best”answers.

4.Enlistasmanypeopleinthepilotasseemsreasonablewithoutwastingyourresources.Probablyfewerpeoplewillbeneededtotesta5-itemquestionnairethana20-itemone.Also,ifyouseethatthesurveyneedslittleimprovement,stop.Aruleofthumbistostartwithfivepeopleandcontinueonwithsmallgroupsuntilyounolongerreceivenewinformation.

5.Forreliability,focusontheclarityofthequestionsandthegeneralformatofthesurvey.Hereiswhattolookfor:

FailuretoanswerquestionsGivingseveralanswerstothesamequestionWritingcommentsinthemarginsofwrittensurveys

Anyoneoftheseisasignalthatthequestionnairemaybeunreliableandinneedofrevision.Arethechoicesinforced-choicequestionsmutuallyexclusive?Haveyouprovidedallpossiblealternatives?Isthequestionnaireorinterviewlanguageclearandunbiased?Dothedirectionsandtransitionsmakesense?Haveyouchosentheproperorderforthequestions?Isthequestionnairetoolongorhardtoread?Doestheinterviewtaketoolong?(Forinstance,youplannedfora10-minuteinterview,butyourpilotversiontakes20minutes.)

6.Tohelpbolstervalidity,youshouldensurethatallrelevanttopicshavebeenincludedinthesurvey(givenyourresources).Forasurveyofpoliticalattitudes,haveyouincludedallpoliticalparties?Controversialissues?Whatelsemustbeincludedforyoursurveytohavecontentvalidity?Ifyouarenotcertain,checkwithotherpeople,includingthetrial-runrespondents.Doesthesurveyhaveroomfortheexpressionofallviews?Supposeyouaresurveyingpeopletofindouthowreligioustheyare.Ifyouhadproofinadvancethatallareveryreligious,youwouldnotneedasurvey.Unlessyoucanshowthatatleastintheoryyoucandistinguishthereligiousfromthenonreligious,noonewouldbelievetheresults.Howdoyoufixthis?Inthetrials,choosepeopleyouknowarereligiousandthoseyouknowarenotandgivethemthesurvey.Dotheirresponsesdiffer?

7.Testyourabilitytogetarangeofresponses.Ifpeopletrulydifferintheirviewsorfeelings,willthesurveycapturethosedifferences?Supposeyoursurveyisofasuburbanneighborhood’sattitudetowardaproposednewhigh-risebuildingdevelopment.Youshouldadministerthesurveytopeoplewhoarebothforandagainstthebuilding.Thisapproachwillhelprevealyourownbiasesinhowquestionsarewordedand,forclosedquestions,mighthelpyouidentifychoicesthatpeoplewhofeelstronglyonewayperceiveasmissingbutthatyoumightnothavethoughtofpreviously.

Considerthis:Inapilotofasurveyofchildren’sself-reporthealthbehaviors,respondentswereasked

howoftentheywashedtheirhandsaftereating.Allsixchildrenbetween8and10yearsofageansweredalwaysafterbeinggiventhechoicesalways,never,andIdon’tknow.Thechoiceswerechangedtoalmostalways,usually,andalmostnever.Withthenewcategories,thesamesixchildrenchangedtheiranswerstotwoalmostalwaysandfourusually.

AFar-ReachingWorld:Surveys,Language,andCultureManysurveysaretranslatedintodifferentlanguages.Ifyouplanontranslatingyoursurvey,donotassumethatyoucanautomaticallyrewordeachquestionintothenewlanguage.Betweentheoriginallanguageandthenextlanguageoftenliesaculturalgap.Youmayneedtorewordeachsurveyquestion.

Toavoidconfusingpeopleandeveninsultingthembecauseyoumisunderstandtheirlanguageorculture,youshouldfollowafewsimpleguidelines.Thesestrategiesinvolveenlistingtheassistanceofpeoplewhoarefluentinthelanguage(anditsdialects)andpilottestingthesurveywithtypicalrespondents.Followtheseguidelines.

GuidelinesforTranslatingSurveysUsefluentspeakerstodothefirsttranslation.Ifyoucan,usenativespeakers.Ifyoucanaffordit,findaprofessionaltranslator.Theartoftranslationisinthesubtleties—wordsandphrasesthattakeyearsandculturalimmersiontolearn.Ifyouusefluentspeakers,youwillminimizethetimeneededtorevisequestionwordingandresponsechoices.Tryoutthetranslatedsurveywiththreetofivenativespeakers.Askthemthesequestions:Whatisthisquestionaskingyou?Canyouthinkofabetterwaytoaskthisquestion?Revisethesurveywiththehelpoftheoriginaltranslator.Translatethesurveybackintotheoriginallanguage.Useadifferenttranslatorforthistask.Doesthis“back-translated”surveymatchtheoriginalversion?Ifnot,thetwotranslatorsshouldworktogethertomakethemmatch.Trytheresultingsurveyonasmallgroup(5–10)oftargetrespondents.Ifthetwotranslatorscouldnotagreeonwording,letthegroupdecide.Revisethesurvey.Pilottestthesurvey.Produceafinalversion.Ifyouwanttofindouttherespondents’backgroundsortheirethnicity,youshouldprobablyrelyonexistingquestionsratherthancreatingyourown.ThefollowingquestionaboutethnicitycomesfromtheU.S.DepartmentofHealthandHumanServices(http://minorityhealth.hhs.gov/templates/content.aspx?ID=9227&lvl=2&lvlID=208).

Example:QuestionAboutEthnicity:U.S.DepartmentofHealthandHumanServicesOfficeofMinorityHealthAreyouHispanic,Latino/a,orSpanishorigin?(Oneormorecategoriesmaybeselected)

1. No,notofHispanic,Latino/a,orSpanishorigin2. Yes,Mexican,MexicanAmerican,Chicano/a3. Yes,PuertoRican4. Yes,Cuban5. Yes,anotherHispanic,Latino,orSpanishorigin

Whatisyourrace?(Oneormorecategoriesmaybeselected)

1. White2. BlackorAfricanAmerican3. AmericanIndianorAlaskaNative

http://minorityhealth.hhs.gov/templates/content.aspx?ID=9227&lvl=2&lvlID=208

4. AsianIndian5. Chinese6. Filipino7. Japanese8. Korean9. Vietnamese10. OtherAsian11. NativeHawaiian12. GuamanianorChamorro13. Samoan14. OtherPacificIslander

ThesequestionsaboutethnicityandracearerelevanttotheU.S.population.IntheUnitedKingdom,othercategoriesareused,suchasWhite,Asian(Indian,Pakistani,Bangladeshi),Black(Caribbean,African),mixedrace,andChinese(http://www.communities.gov.uk/publications/corporate/statistics/citizenshipsurvey200809equality).

SummingUpThelengthofasurveydependsonwhatyouneedtoknowandhowmanyquestionsarenecessarysothattheresultinganswersarecredible.Anotherconsiderationistherespondents.Howmuchtimedotheyhaveavailable,andwilltheypayattentiontothesurvey?Questionsshouldproceedfromthemostfamiliartotheleast.Placerelativelyeasy-to-answerquestionsattheendofasurvey.Avoidmanyitemsthatlookalike.Avoidskippatternsinwrittenself-administeredquestionnaires.Onlinesurveysareeffectivevehiclesforbranchingifyoumustusethembecauseyoucandesignthesoftwaretoguidetherespondent,andeverycommercialvendorprovidesskip-patternlogic.Thekeytoagoodtelephoneorface-to-faceinterviewistraining,whichshouldensurethatallinterviewersknowwhatisexpectedofthemandthattheyaskallthequestionsinthesameway,withinthesameamountoftime.Interviewsshouldbemonitoredtoensureinterviewersareaskingquestionsinthesamewayandmaximizeconsistencyofresponses(reliability).Pilottestingbolstersreliabilityandvaliditybecauseitcanhelpyouseethatalltopicsareincludedandthatsufficientvarietyintheresponsesisavailable.Areliablesurveyprovidesconsistentinformation.Avalidsurveyprovidesaccurateinformation.Onewaytoensurethatyouhaveareliableandvalidsurveyistouseonethatsomeoneelsehaspreparedanddemonstratedtobereliableandvalidthroughcarefultesting.Test-retestreliabilityisusuallycomputedbyadministeringasurveytothesamegroupontwodifferentoccasionsandthencorrelatingthescoresfromonetimetothenext.Equivalencereliabilityisdeterminedbygivingtwoormoreformsofthesamesurveytothesamegroupofpeopleonthesamedayorbygivingdifferentformsofthesurveytotwoormoregroupsthathavebeenrandomlyselected.Anothermeasureofreliabilityishowinternallyconsistentthequestionsonasurveyareinmeasuringthecharacteristics,attitudes,orqualitiesthattheyaresupposedtomeasure.Totestforinternalconsistency,calculateastatisticcalledcoefficientalpha(orCronbach’salpha).Predictivevaliditymeansthatthesurvey’sdataforecastrespondents’abilitytoperformagiventaskorbehaveinacertainway.Concurrentvaliditymeansthatthedatafromanewsurveycorrelatetothedatafromapreviouslyestablishedone.Asurveyhascontentvalidityifexpertsagreethatitsitemsorquestionsaccuratelyrepresentthe

http://www.communities.gov.uk/publications/corporate/statistics/citizenshipsurvey200809equality

characteristicsorattitudestheyareintendedtomeasure.Constructvalidityisestablishedexperimentallybytryingthesurveyonpeoplewhomclinicalexpertssaydoanddonotexhibitthebehaviorassociatedwiththeconstruct.Ifyouplanontranslatingasurvey,enlisttheassistanceofpeoplewhoarefluentinthelanguage(anditsdialects)andpilottestthesurveywithtypicalrespondents.Choosepublishedsurveyswithcare.Checktoseethattheyarereliable,valid,andappropriateforthepeopleinyoursurvey.

ThinkAboutThis1. Locatethearticlesthatfollowandanswerthesequestions:

1. Namethesurveyorsurveysdiscussedinthearticle.2. Describethepurposeofeachsurvey.3. Discussthereliabilityandvalidityofeachsurvey.4. Areyousatisfiedthatthesurveysarereliableandvalid?Explain.

2. Selectanythreesurveysthatinterestyou,andanswerthesamequestionsasinQuestion1.3. Readthecasestudybelow.Isitokayforthethirdprofessortohaveaccesstothesurveyevenifheis

responsibleformailingtherespondentstheirincentivechecks?

ThreeprofessorsreceiveagrantfromtheNationalLibraryofEducationtodoanonlinesurveyof60schoolstofindoutaboutthesuccessoftheirdataexchangesystem.Eachprofessorteachesatadifferentcollege.TwooftheprofessorsreceivepermissionfromtheirrespectiveIRBstogainaccesstorespondents’emailaddressesinordertosendoutthesurvey,followupwithnonrespondents,andperformthestatisticalanalysis.Thethirdprofessordoesnothaveexplicitpermissiontodoanyofthesethings,butheisinchargeofsendingoutthefinancialreimbursementsforparticipationinthesurvey.ThethreeprofessorsusedtheE-LineOnlineSurveyCompanytobuildtheirsurvey,sowiththeproperusernameandpassword,anyone,includingthethirdprofessor,canlogontothesurveysiteandviewtheresultsbyindividualandbyquestion.Thethirdprofessorneedstoobtainemailaddressestoaskformailingaddresses,whicharenotincludedinthesurvey.Helogsontothesurveyandrecordstheemailaddresses.HedoesnotlookatanyresponsestothesurveybecausehedoesnothaveIRBclearancetodoso.Heemailstheparticipantsand,immediatelyuponreceivingtheirmailingaddresses,sendsoutthereimbursementchecks.

ReferencesAsgeirsdottir,B.B.,Sigfusdottir,I.D.,Gudjonsson,G.H.,&Sigurdsson,J.F.(2011).Associationsbetweensexualabuseandfamilyconflict/violence,self-injuriousbehavior,andsubstanceuse:Themediatingroleofdepressedmoodandanger.ChildAbuse&Neglect,35(3),210–219.

Bauer,D.L.,Whitman,L.A.,&Kosson,D.S.(2011).Reliabilityandconstructvalidityofpsychopathychecklist.CriminalJusticeandBehavior,38(10),965–987.

Dunn,K.I.,Mohr,P.,Wilson,C.J.,&Wittert,G.A.(2011).Determinantsoffast-foodconsumption.Anapplicationofthetheoryofplannedbehaviour.Appetite,57(2),349–357.doi:10.1016/j.appet.2011.06.004.

Flynn,K.E.,Smith,M.A.,&Freese,J.(2006).WhendoolderadultsturntotheInternetforhealthinformation?FindingsfromtheWisconsinLongitudinalStudy.JournalofGeneralInternalMedicine,21(12),1295–1301.

Graham,J.M.(2011).Measuringloveinromanticrelationships:Ameta-analysis.JournalofSocialand

PersonalRelationships,28(6),748–771.

Holman,E.A.,Silver,R.C.,Poulin,M.,Andersen,J.,Gil-Rivas,V.,&McIntosh,D.N.(2008).Terrorism,acutestress,andcardiovascularhealth:A3-yearnationalstudyfollowingtheSeptember11thattacks.ArchivesofGeneralPsychiatry,65(1),73–80.

O’Connor,T.M.,Yang,S.-J.,&Nicklas,T.A.(2006).Beverageintakeamongpreschoolchildrenanditseffectonweightstatus.Pediatrics,118(4),e1010–e1018.

Overby,M.,Carrell,T.,&Bernthal,J.(2007).Teachers’perceptionsofstudentswithspeechsounddisorders:Aquantitativeandqualitativeanalysis.Language,Speech,&HearingServicesinSchools,38(4),327–341.

Richardson,J.,Coid,J.,Petruckevitch,A.,Chung,W.S.,Moorey,S.,&Feder,G.(2002).Identifyingdomesticviolence:Crosssectionalstudyinprimarycare.BritishMedicalJournal,324(7332),274.

Skoffer,B.(2007).Lowbackpainin15-to16-year-oldchildreninrelationtoschoolfurnitureandcarryingoftheschoolbag.Spine,32(24),E713.

Wolfson,M.,Champion,H.,Rogers,T.,Neiberg,R.H.,Barker,D.C.,Talton,J.W.,etal.(2011).Evaluationoffreetogrow:Headstartpartnershipstopromotesubstance-freecommunities.EvaluationReview,35(2),153–188.

4SAMPLING

OverviewShouldyousurveyeveryoneorjusttakeasample?Theanswerdependsonhowquicklyyouneedresults,ifthecredibilityofthefindingswillsufferifsomeoneorsomegroupisleftout,andyourfinancialandtechnicalresources.

Samplingcanbedividedintotwocategories.Arandomsampleisselectedbyanobjectivemethod(suchasdrawingnamesatrandomfromahat),andyoucanalsocalculateeachperson’schancesofselection.Nonrandomsamplesareconvenient:Youselectonlythoserespondentswhoarenearbyandwillingandavailabletocompletethesurvey.

Howlargeshouldasamplebe?Relativelylargersamplesreducesamplingerrors.Ifyouwanttoevaluatetheperformanceoftwo(ormore)groups,statisticalmethodscanguideyouinselectingsamplesizeslargeenoughtodetectdifferencesbetweenthegroups—ifdifferencesactuallyexist.Statisticalmethodsmayalsotellyouhowlargeadifferenceyoucanobserve(andifyouwillbeabletoobservedifferences)withthesamplesizeyouhave.

Theresponserateconsistsofthenumberofcompletedsurveysdividedbythenumberofsurveyseligibleforcompletion.Toimproveresponserate,makesurethatrespondentscaneasilycompleteandsubmitthesurvey,trainsurveyorstoadministersurveysandfollowup,monitorthequalityofsurveyadministration,keepresponsesconfidential,andproviderespondentswithincentivesandrewardswhenpossibleandethical.

SampleSizeandResponseRate:WhoandHowMany?Whenyouconductasurvey,doyouhavetoincludeeveryone?Theanswerisprobablynot.Often,thetargetpopulation—everyone—consistsofalargenumberofpeople.Surveyingthemisnotpractical.

Ifyoudecidetosample,youmustaskthefollowing:Howmanypeopleshouldbeincluded?Ifyourcompanyhas1,000employeesandyouwanttosurveyjustsomeofthem,howdoyoudecidehowmanypeopletoinclude?Sayyouwanttocomparelong-andshort-termjobsatisfaction.Statisticalmethodscanhelpyoudecidehowmanypersonsshouldbeincludedeachtime(shortversuslongterm)sothatifadifferenceinsatisfactionactuallyoccursovertime,youwillbeabletodetectit.Sayyouhave3monthstoconductthesurveyandyouthinkyoucansurveyabout300ofthecompany’semployees.Statisticalmethodscanhelpyoudeterminehowmuchofaneffectyoucandetect,giventhesizeofthesampleyourresourcesallowyoutoassemble.

Supposeyouwanttofindoutwhetheryourneighborswillsupportacommunityvigilanceprograminwhicheachhouseholdtakesresponsibilityforwatchingatleastoneotherhousewhentheownersareaway.Consideralsothatyoudefineyourcommunityashaving1,000homes.Doyouneedtoincludeallhouseholds?Ifyoudo,willtheprogrambemorelikelytoworkthanotherwise?Herearesomequestionsyoushouldanswer.

1. Howquicklyaredataneeded?Supposearecentincreaseinhouseburglariesisthemotivationforthesurveyandyouwanttogetstartedimmediately.Ifyouwaittosurveyall1,000homesinyourneighborhood,youmaywasteprecioustime.

2. Whattypeofsurveyisplanned?Ifyouaregoingtouseatelephoneoronlinesurvey,yoursurveywillprobablytakelesstimethanifyouplantointerviewpeopleintheirhomesorwaitforthemtoreturntheirsurveysinthemail.

3. Whatareyourresources?Iftheyarelimited,youhavetoselectasurveymethodsuchastelephoneinterviewingratherhomeinterviewing.

4. Howcrediblewillyourfindingsbe?Ifall1,000homesparticipate,thenyouwillhavenoproblemarguingthatyoursurveyisrepresentativeoftheneighborhood.Ifonly10homesparticipate,youwillrunintoacredibilityproblem.

5. Howfamiliarareyouwithsamplingmethods?Samplingmethodscanberelativelysimpleorcomplex.Nationalpollsandlargeresearchstudiesusesophisticatedtechniquesthatdependontheskillsofstatisticiansandothertrainedexperts.Othermethodsmaybeeasiertoimplement,buttheresultingsamplemaynotbeanaccuratereflectionofthelargerpopulation.

Howdoyouselectasample?Youcanusearandomorconveniencesamplingmethod.Arandomsampleisselectedobjectively.Anexamplewouldbetheuseofspecialsoftwaretorandomlyselectwinninglotterynumbers.Statisticalmethodsareavailabletocalculatetheprobabilitythateachpersonhasofbeingchosen(orwinningthelottery).Aconveniencesampleincludespeoplewhoareavailableandwillingtotakethesurvey.Thesampleselectionprocessisnotconsideredobjectivebecausenoteveryeligiblepersonhasanequalchance.Ifyouareillonthedayofthesurvey,youwillnotbeabletoparticipateevenifyoumeetallotherrequirements.

Considerthesetwocases.

Example:RandomandConvenienceSamplingCase1.AsurveyaimstofindouthowteachersintheLoebSchoolfeelaboutcertainschoolreforms.All100teachers’namesareputintoahat,thenamesarejumbled,andtheprincipalselects50forthesurvey.

Case2.AsurveyisconductedtofindoutLosHadasSchoolDistrictteachers’viewsoncertainschoolreforms.Tenteachersarechosentobeinterviewedfromeachofthedistrict’ssixelementaryschools,15areselectedfromitsfourintermediateschools,and60arechosenfromitsonehighschool.Participatingteachersarevolunteerswhowererecommendedbytheirprincipal.Theymeetthefollowingcriteria:Theyhavebeenteachinginthedistrictfor5yearsormore,theybelongtooneofthreeteachers’associationsorunions,andtheyhaveparticipatedinatleastonemeetingduringthepastyearonthedistrict’sschoolreform.

Inthefirstsurvey,asampleof50of100teachersischosenfromahat.Thistypeofsamplingisconsistentwithrandomsamplingbecausetheselectionmethodisobjectiveandyoucanuseamathematicalformulatocalculatetheprobabilityofeachpersonbeingselected.Also,youcanusuallyexpectthatthetwogroupsofteacherswillnotbesystematicallydifferentfromoneanother.Ineachgroupof50,youarelikelytohavesimilarnumbersofpeoplewhoaresmart,healthy,andgenerousandsimilarnumberswhoarenotsmart,unhealthy,andmiserly.

Inthesecondsurveyexample,principalsmakerecommendationsandeligibleteachersvoluntarilychoosetoparticipate.Principalsmayhavetheirfavorites.Teacherswhovolunteermaysystematicallydifferfromthosewhodonot.Theymaybemoreenthusiasticaboutthesurveyorhavemoretimetocompleteit,forexample.Thissamplingstrategyisnotobjective.Itcomesupwithanonrandomsample.

RandomSamplingMethodsThreeofthemostcommonlyusedrandomsamplingmethodsarethefollowing:

Simplerandomsampling

StratifiedrandomsamplingSimplerandomclustersamplingStratifiedrandomclustersampling

Asimplerandomsampleisoneinwhicheachpersonhasanequalchanceofbeingselectedfromapopulation.Thepopulationcontainseveryonewhoiseligibleforthesurvey.

Thefollowingisasimplerandomsampling.

Example:SimpleRandomSampling(1)Youwanttoselect100peoplefromphilanthropicfoundationstosurveythemaboutthetypesofgrantstheysponsor.Atotalof400peoplecanprovidethisinformation.Youplacetheirnamesinanyorder.Eachnameisgivenanumberfrom001to400.Then,usingatableofrandomnumbers(foundinstatisticstextbooks),youselectthefirst100peoplewhosenumbersshowuponthetable.

Eachpersoninthisscenariohasanequalopportunityforselection.Thepopulationconsistsof400people.Chancealonedecideswhichofthemissampled.

Thefollowingisnotrandomsampling.

Example:NotRandomSamplingYouwanttosample100peoplefromphilanthropicfoundationstosurveythemaboutthetypesofgrantstheysponsor.Atotalof400peoplecanprovidethisinformation.Youselect25peopleineachoffourareasofthecountry.

Somepeopleinthisscenariohavenochanceofselection:thosewhodonotliveinyourfourchosengeographicareas.

Hereisanotherexampleofsimplerandomsampling.

Example:SimpleRandomSampling(2)Twohundrednurses,therapists,andsocialworkersemployedbyaMidwestcitysignedupforanadultdaycareseminar.Thecityonlyhadenoughmoneytopayfor50participants.Theseminardirectorusedtherandomassignmentfeatureinherstatisticsprogramsoftwaretoselect50names.

Tofacilitatesimplerandomsamplingfortelephonesurveys,somesurveyorsuseatechniquecalledrandomdigitdialing.Inoneofitsvariations,calledtheplus-oneapproach,adigitisaddedtothetelephonenumberthatisactuallyselected.Iftheselectedtelephonenumberis(311)459-4231,thenumbercalledis(311)459-4232.Thistechniquehelpstomakeupforthefactthatinmanyareasofthecountry,particularlyinurbanareas,peopledonotlisttheirtelephonenumbers.

MakingtheDecision

Whenshouldyouchoosesimplerandomsampling?Theadvantagesofsimplerandomsamplingarethese:

Itisthesimplestofallrandomsamplingmethods.Aidsareavailabletoassistyou.Moststatisticstextbookshaveeasy-to-usetablesfordrawingarandomsample.Analternativeistousetherandom-numberfeaturefoundinallstatisticalsoftware.

Amajordisadvantageofsimplerandomsamplingisthatitcannotbeusedtodividerespondentsintosubgroupsorstrata(e.g.,60%maleand40%female).Tomakesureyouhavetheproportionsyouneedinasample,youneedtostratify.

StratifiedRandomSamplingInsimplerandomsampling,youchooseasubsetofrespondentsatrandomfromapopulation.Instratifiedrandomsampling,youfirstsubdividethepopulationintosubgroupsorstrataandselectagivennumberorproportionofrespondentsfromeachstratumtogetasample.

Youcan,forexample,usestratifiedrandomsamplingtogetanequalrepresentationofmalesandfemales.Youdothisbydividingtheentiregroupintosubgroupsofmalesandfemalesandthenrandomlychoosingagivennumberofrespondentsfromeachsubgroup.Thismethodofsamplingcanbemoreprecisethansimplerandomsamplingbecauseithomogenizesthegroups,butonlyifyouchoosethestrataproperly.Thatis,donotsamplemenandwomenunlessyouareplanningtomakecomparisonsbetweenthem.Youshouldplantomakecomparisonsonlyifyouhavesomereasonstobelieve,inadvance,thatthosecomparisonsmightbemeaningful.Inasurveyofvoterpreference,forexample,ifyouhavesomeevidencethatmenandwomenvotedifferently,thenitmakessensetobesurethatyoursurveyincludesenoughmalesandfemalestocomparethem.Withrandomsamplingalone,youmightfindthatbychanceyouhaveasurveysamplethatconsistsmainlyofmenormainlyofwomen.

Hereishowstratifiedrandomsamplingworks.

Example:StratifiedRandomSamplingTheUniversityHealthCenterisconsideringtheadoptionofanewprogramtohelpyoungadultsloseweight.Beforechangingprograms,theadministrationcommissionedasurveytofindout,amongotherthings,howtheirnewprogramcomparedwiththecurrentoneandhowmaleandfemalestudentsofdifferentagesperformed.Previousexperiencehadsuggestedthatolderstudentsappearedtodobetterinweightreductionprograms.Therefore,thesurveyorsplannedtogetasampleofmenandwomenintwoagegroups—17to22yearsand23to28years—andcomparetheirperformanceineachoftheprograms.

About310undergraduatessignedupforthehealthcenter’sregularweightreductionprogramforthewinterseminar.Ofthe310participants,140werebetween17and22yearsold,and62oftheseweremen.Some170studentswerebetween23and28years,and80oftheseweremen.Thesurveyorsrandomlyselected40personsfromeachofthefoursubgroups(male,female,ages17–22,andages23–28)andrandomlyassignedeveryotherstudenttothenewprogram.Thesamplelookedlikethis:

MakingtheDecisionAnadvantageofstratifiedrandomsamplingisthatthesurveyorcanchooseasamplethatrepresentsthevariousgroupsandpatternsofcharacteristicsinthedesiredproportions.

Thedisadvantagesofstratifiedrandomsamplingarethatitrequiresmoreeffortthansimplerandomsamplinganditoftenneedsalargersamplesizethandoesarandomsampletoproducestatisticallymeaningfulresults.Remember,foreachstratumorsubgroup,youprobablyneed20to30personstomakemeaningfulstatisticalcomparisons.

Ifyouhavedifficultyselectingastratifiedrandomsample,keepinmindthatthesameincreaseinprecisionobtainedwithstratificationcangenerallybeproducedbyincreasingthesamplesizeofasimplerandomsample.Increasingsamplesizemaybeeasierthanimplementingastratifiedrandomsample.

SimpleRandomClusterSamplingSimplerandomclustersamplingisusedprimarilyforadministrativeconvenience,nottoimprovesamplingprecision.Sometimesrandomselectionofindividualssimplycannotbeused.Forinstance,itwouldinterrupteveryhospitalwardtochoosejustafewpatientsfromeachwardforasurvey.Sometimesrandomselectionofindividualscanbeadministrativelyimpossible.

Onesolutiontotheproblemofusingindividualsasasamplingunitistousegroupsorclustersofrespondents.

Insimplerandomsampling,yourandomlyselectasubsetofrespondentsfromallpossibleindividualswhomighttakepartinasurvey.Clustersamplingisanalogoustorandomsampling,exceptthatgroupsratherthanindividualsareassignedrandomly.Thismethodpresupposesthatthepopulationisorganizedintonaturalorpredefinedclustersorgroups.Hereishowitworks.

Example:SimpleRandomClusterSamplingTheCommunityMentalHealthCenterhas40separatefamilycounselinggroups,eachwithabout30participants.Thecenter’sdirectornoticedadeclineinattendanceinthelastyearanddecidedtotryoutanexperimentalprograminwhicheachindividualwouldbetestedandinterviewedseparatelybeforebeginningtherapy.Theprogramwasexpensiveandthecentercouldonlyaffordtofinancea150-personprogramatfirst.

Randomlyselectingindividualsfromallgroupmemberswouldhavecreatedfrictionanddisturbedtheintegrityofsomeofthegroups.Instead,asimplerandomclustersamplingplanwasused,inwhich5ofthe30-membergroups—150peoplealltogether—wererandomlyselectedtotakepartintheexperimentalprogram.Eachgroupwastreatedasacluster.Attheendofthe6months,theprogressoftheexperimentalprogramwascomparedwiththatofthetraditionalone.

MakingtheDecisionTheadvantagesofsimplerandomclustersamplingarethese:

Itcanbeusedwhenselectingindividualsrandomlyisinconvenientorunethical.Itsimplifiessurveyadministration.

Thedisadvantageofsimplerandomclustersamplingisthatitrequirescomplexstatisticalmethodstoreconcilesamplingunits(thehospital,street,school)andanalyticunits(patients,homeowners,students).Thatis,youaresamplingbycluster(suchasschools),butyouareanalyzingdatafromindividuals(suchasstudents).

Althoughintheexampleyouhave150peopleinthesurvey,youreallyhavejust5units(the5groupsof30personseach)tostudy.Whycan’tyoustudyeachofthe150personsindividually?Whenpeopleareinspecialgroups—classes,clubs,organizations,neighborhoods—theytendtosharesimilarcharacteristicsandviews.Studyingeachindividualmayberedundantbecauseonepersonmaybesimilartothenext.Youneedarelativelylargenumberofpeopleforeachclusterforthisproblemtobecomelessnoticeable.Justhowlargeanumberyouneedisastatisticalissue.

Complexsamplingstrategiesrequireanunderstandingofsamplingstatistics.

SystematicSamplingInsystematicsampling,youpickanumber,say5,andselectevery5thnameonalistthatrepresentsthepopulation.Ifalistcontains10,000namesandthesurveyorwantsasampleof1,000,heorshemustselectevery10thnameforthesample.

Supposeyouhavealistof500namesfromwhichyouwanttoselect100people.Youcanrandomlyselectanumberbetween1and10.Ifyouchosethenumber3,youwouldbeginwiththe3rdnameonthelistandcountevery5thnameafterthat.Yoursampleselectionwillresultinthe3rdname,8th,13th,andsoonuntilyouhad100names.Ifyouselectthe“start”numberatrandom,systematicsamplingresemblessimplerandomsampling.

MakingtheDecisionThereisadangerinsystematicsampling.Listsofpeoplearesometimesarrangedsothatcertainpatternscanbeuncovered;ifyouuseoneoftheselists,yoursamplewillbesubjecttoabiasimposedbythepattern.Forinstance,relativelyfewpeopleintheUnitedStateshavelastnamesbeginningwithW,X,Y,andZ,andtheymaybeunderrepresentedinasystematicsample.Hereisanotherexample.Supposeyouaresamplingclassroomssothatyoucansurveystudentstofindoutabouttheirattitudestowardschool.Sayalsothattheclassroomsarearrangedinthisorder:

Supposefurtherthatyouselecteverythirdclassstartingwith1a.Thesamplewillconsistofclassrooms1a,4a,7a,andsoon,toNa.Thesurveyofattitudestowardschoolcanbebiasedifeach“a”correspondstoalocationwithintheschoolthatfacesthelawnandisquietandthe“b”and“c”classroomsfacethesportsarenaandarenoisy.

Inconsideringtheuseofsystematicsampling,carefullyexaminethelistofpotentialrespondentsfirst.Ifyoususpectbiasbecauseoftheorderorplacementofsamplingunits(people,classrooms),useanothersamplingmethod.

ConvenienceSamplesAconveniencesampleisonethatyougetbecausepeoplewhoarewillingtocompletethesurveyarealsoavailablewhenyouneedthem.Sayyouwanttofindoutwhetherthestudenthealthservicewasanygood.Youplantointerview50students.Ifyoustandneartheclinicentranceduringtheday,youcanrecruiteachpersonwalkingin.Whenyouhaveacompletesetof50interviews,youhaveaconveniencesample.Hereareseveralsourcesofbiasinthissample:

Studentswhoarewillingtobeinterviewedmaybemoreconcernedwiththehealthservicethanthosewhorefuse.Studentswhousetheserviceatthetimeofyourinterviewmaybegoingforconvenience;sickerstudentsmayusetheserviceatnight.Studentswhotalktoyoumayhaveagripeandwanttocomplain.Studentswhotalktoyoumaybethemostsatisfiedandwanttobrag.Studentsmaywanttotalkbutmayhavenotimeatthemoment;thesemaybeworkingstudents.Perhapsworkingstudentsaredifferentfromotherstudents(older?)intheirneedsandviews.

MakingtheDecisionBecauseofbias,conveniencesamplesareunconvincingunlessyouproveotherwise.Here’showyoumightimproveonthecredibilityofyourconveniencesample:

Askrefusersandparticipantshowconcernedtheyarewiththeirhealthandcomparetheresponses.Youmayfindnodifferences;ifso,thenyourconveniencesample’sfindingsaresupported.Visittheclinicatnighttofindoutwhetherstudentsusingthehealthservicethenaredifferentfromthedaystudentsintheirhealthstatus.Youmayfindnodifferences.Again,ifso,thenyoursampleissupported.Askstudentsiftheyhaveagripe.Askstudentsabouttheirsatisfaction.Askstudentswhytheyrefusetoparticipate.Isitbecausetheypresentlydonothavethetime?Comparestudentswhoparticipatewiththosewhodonotintermsofage,gender,andsoon.Ifnodifferencesarefound,yourconveniencesample’sfindingsareeasiertosupportthanotherwise.If

differencesarefound,youneedtorethinkyoursurveymethodology.

OtherConvenienceSamplingMethodsConsiderthesecommonsituations.

Example:OtherConvenienceSamples1. Asurveyof100deansoflawschools,seniorpartnersinlargelawfirms,andjudgesisconductedtofindouttowhich

lawyerstheygotosolvetheirownlegalproblems.TheresultsarepublishedinGlobalNewsandWorldReportsunderthetitle“TheWorld’sBestLawyers.”

2. Whatmakescollegestudentsliberalorconservative?Familybackground?Regionofthecountryinwhichtheywereborn?Currentreligiouspractices?Educationalattainment?Income?AsurveyisconductedofmembersoftheYoungConservativeAssociationandtheYoungLiberalSociety.Anassessmentoftheresultsrevealsthereasonsforstudents’views.

3. Apopularmoviedownloadingservicehassubstantiallyraiseditsmonthlyfee.Howwilltheincreasedpriceaffectsubscribers?Willmostremainwiththeserviceorgosomewhereelse?Atotalof80,000peoplearecontactedtocompleteanonlinesurvey.AllpotentialrespondentshaveparticipatedpreviouslyinsurveysconductedbySurveyPanels,anInternetsurveycompany.

Inthefirstexample,the“top”lawyersmayprovidetheservicesneededbydeans,seniorlawfirmpartners,andjudges.Howapplicablearethetoplawyers’servicestotypicallegalproblems?Inthesecondexample,studentsinonlytwoorganizationsarebeingsurveyed.Canwetrustthattheyrepresentallstudents,includingthosewhobelongtoothergroupsorchoosenottojoinany?Inthethirdexample,thesurveyissentto80,000peoplewhohaveparticipatedinotheronlinesurveysconductedbySurveyPanels.AllrespondentsarewillingtocompleteanInternetsurvey.Dotheyrepresenttheaverageuserofthefilmdownloadingservice?

Youuseconveniencesamplesbecausetheyareavailableandaccesssavesthetimeandexpertiseneededtodevelopasamplingplanandimplementit.Conveniencesamplesareusefulifthesurveyoriscertainthattheresponsesapplytothetargetpopulation.Theybecomeproblematiciftheresponsescomefrompeoplewhodiffersubstantiallyinimportantways(e.g.,age,gender,Internetsavvy)fromthetargetpopulation.

Thesearestandardconveniencesamplingtechniques:

Snowballsamples

Previouslyidentifiedmembersofagroupidentifyothermembers.Forexample,youselectCEOsandaskthemtonominateothers.

Quotasamples

Thegroupisdividedintosubgroupsinspecificproportions.Thisissimilartostratifiedsampling.

Purposivesamples

Apurposivesample,alsocommonlycalledajudgmentalsample,isonethatisselectedbasedontheknowledgeofapopulationandthepurposeofthestudy.Therespondentsareselectedbecauseofsomecharacteristic.Supposeyouareinterestedinfindingoutwhypeopleshopinaparticularstore.Youmightspendadayinterviewingshoppers.Or,youmightinterviewpeoplewhodonotplantoshopinthatstore.Inthiscaseyouareusingapurposivesamplebecauseyouwantinformationonthecharacteristicsofpeoplewhodo(ordonot)shopinthestore.

Focusgroups

Tento20peoplearebroughttogethertoanswerspecificquestions.Atrainedleaderconductsthesessions.Atranscriberisusuallypresenttosummarizethediscussion.

Panels

Peoplearerecruitedtoparticipateinsurveysonanas-neededbasis.Manyonlinesurveycompaniesrecruitpanelsofrespondentsfromthemillionsofpeoplewhoparticipateintheirsurveys.Large,reputablecompaniesscreenrespondentssothattheyhaveaccesstoadiversegroupofpeople,say,forinstance,savvysmartphoneusers,workingmoms,videogameplayers,orarandomselectionofUnitedStatesadults.Usually,thesurveyordesignsthequestionnaireandspecifieswhoshouldbeincludedinthesurvey.Thesurveycompanydoestherest.Paymentisbyresponse.Forinstance,ifthesurveyorneeds300responses,andthesurveycompanycharges$3.00foreachresponse,thenthecostofthesurveyis$900.Itusuallytakesaboutaweektogetacompletesetofresponses.Toencourageresponses,surveycompaniesusuallyprovideincentivestothepanelmembers.Forexample,onepopularonlinecompanydonates$0.50toacharitychosenbytheindividualpanelistandalsogiveshimorherchancetowina$100giftcard.Thecompanydoesnotprovidedirectpaymentorcreditssoastodiscouragesurveytakersinterestedonlyincollectingrewards.

FindingtheSample:WhoIsIn?WhoIsOut?Howdoyoufindasample?Beforeyoulook,youmustdecidewhoshouldbeincluded(andexcluded)fromthesurvey.SupposeyouwanttoevaluatetheeffectivenessofCOMPULEARN,anonlineprogramtokeepemployeesuptodateonhowtomarketproductselectronically.Youdecidetosurveyasampleofusersandsetthesestandardsforrespondentstobeincludedinthesurvey:

Participateintheprogramfor1monthormoreBewillingtolearntouseFREENetsoftware(about30minutes)Be30yearsofageorolderReadEnglish

Theseeligibilitycriteriahelpyounarrowyoursample.Buttheydosomethingelseaswell:Theyrestrictyourfindingsjusttootherpeoplewhoalsomeetthestandards.Here’swhy:Afteryoursurveyiscomplete,youwillbeabletotellabouttheeffectivenessofCOMPULEARNonlyforpersonswhohavebeenintheprogramfor1monthormore,whocanusethespecifiedsoftware,whoare30yearsofageorolder,andwhoareEnglishspeaking.Apersonwhois29andmeetsallotherstandardsis,bydefinition,ineligibleandyouwillnotbeabletoapplythesurvey’sfindingsto29-year-olds.(Afterall,youchose30yearsofageasaneligibilitycriterionbecauseitpresumablymeantacriticaldemarcationpoint.)

Everytimeyoudoasurvey,youmustdecidethis:DoIwanttosampleeveryoneorjustpeopleorplaceswithcertaincharacteristics?Usuallytheansweris“justpeopleorplaceswithcertaincharacteristics(suchasschoolswith350ormorestudents).”

HowLargeShouldYourSampleBe?Somesurveystakeplacewithjustonegroup.Apollofasampleofvotersisthistypeofsurveysample.Thetrickistoselectasamplethatisrepresentativeofallvoterswhointerestyou.Othersurveysareoftwoormoregroups.Forexample,asurveycomparingthecareerplansofstudentsintheJOBSprogramwithstudentsintheCAREERprogramisthissecondsample.Whencomparingstudents’careerplans,youmaywantrepresentativesamplesofstudentsinbothgroups.Inaddition,youhavetothinkaboutthenumberofstudentsyouneedineachofthetwogroupssothatifadifferenceexistsbetweenthemyouhaveenoughpowertodetectit.

Considertheseexamples.

1. Onegroup,nointerventionorprogram:Youworkfortheparksystemandwanttofindoutwhereyoungpeopledotheirmountaincycling.

Objective:Tosurveyyoungmountainbicyclists’preferencesPopulation:Mountainbicyclists21yearsandyoungerSamplingquestion:Howmanymountainbicyclistsshouldbesurveyedtomakesurethatthesampleisafairrepresentationofmountainbikeriders21yearsandyounger?

2. Twoormoregroupsandanintervention:Youhavetwogroups.Youwanttocomparethemfordifferencesafteroneofthetwohasbeenpartofanewactivity.

Objective:ToevaluateachessprogramPopulation:ChildrenwhoareinanexperimentalchessprogramandacontrolgroupofchildrenwhoarenotSamplingquestion:Howmanychildrenhavetobeineachgrouptodetectapositivedifferenceifoneoccurs?

Whenyouthinkaboutsamplesize,youmustalsothinkaboutthestandarderror,astatisticusedtodescribesamplingerrors.Errorexistsbecause,whenyousample,youselectfromalargerpopulationandthesampletypicallydiffersfromthepopulation.Thisdifferenceisarandomresultofsampling.Youcancontrolitbutprobablynoteliminateitentirely.Ifyoudrewaninfinitenumberofsamples,themeansoraverageswouldformadistributionthatclustersaroundthetruepopulationmean.Thisdistribution,whichhasabellshape,istheso-callednormaldistribution.Ingeneral,largersamplesaremorelikelytocollectaroundthetruepopulationmeanandbeamoreaccurateestimationofthepopulationmean.

TheStandardErrorLargersamplestendtoreducesamplingerrorswhenthesamplesarerandomlyselected.Thestatisticusedtodescribesamplingerroriscalledthestandarderrorofthemean.Itisthestandarddeviationofthedistributionofsampleestimatesofmeansthatcouldbeformedifaninfinitenumberofsamplesofagivensizeweredrawn.

Trythis:Inasurveyof300randomlyselectedrespondents,70%answeryestothequestion,“Ismountain

bikingoneofyourfavoritesports?”Thesurveyorreportsthatthesamplingerroris9.2percentagepoints.Ifyouadd9.2to70%,orsubtract9.2from70%,yougetaninterval—calledaconfidenceinterval—thatrangesbetween60.8%and79.2%.Usingstandardstatisticalmethods,thismeansthatthesurveyorcanestimatewith95%confidence(the95%isaconvention)thatthe“true”proportionofmountainbicyclerswhoansweryesfallswithintheinterval.

Thetrickiskeepingtheconfidenceintervalsmall.Inpractice,largersamplesusuallyreducesamplingerrorsinrandomsamples.Butaddingtothesamplereducestheerroragreatdealmorewhenthesampleissmallthanwhenitislarge.Also,differentsamplingmethods(suchassystematicsampling)mayhavedifferenterrorratesfromrandomsampling.

Rememberthatnotallerrorscomefromsampling.Althoughyouwantalargeenoughsampletokeeptheerrorlow,youdonotwantsamplesizepressurestodistractyousothatothersourcesoferrorblightyoursurvey.Othersourcesoferrorincludeambiguouseligibilitycriteria,badlydesignedandadministeredsurveys,andpoorreturns.

StatisticalMethods:SamplingforTwoGroupsandanInterventionSupposeyouwanttocomparetwogroups.First,dividethepopulationintwo(say,randomly).Then,usestatisticalcalculations(suchastheonesthatfollow)tofindoutwhethereachgroup’ssamplesizeislargeenoughtopickupadifference,ifoneispresent.Ifthepopulationislarge,youmaywanttoselectasample(say,atrandom)andthenassignpersonstothetwogroups.Ifyouselectasampleatrandom,youhaverandomsampling.Ifyouassignpeopletogroupsatrandom,youhaverandomassignment.Ifyouselectallfiveschoolsinacityandrandomlyassignallstudentsineachoftheschoolstogroups,youhavenonrandomclustersamplingandrandomassignment.If,however,yourandomlyselectfiveschoolsinacity,assignthreetoanexperimentandtwotoacontrol,andputallstudentsintheexperimentalschoolsintheexperiment,youhaverandomclustersamplingandnonrandomassignment.

Usethefollowingchecklisttogetorevaluateasamplesizewhenyouhavetwogroupsandanintervention.

Example:SampleSizeCalculationsforSamplingTwoGroupsandanInterventionAssembleandclarifysurveyobjectivesandquestions.

Decidethesurvey’spurposes.Considerthese:

Survey1:QualityofLife

Objective:TodeterminewhetheryoungerandolderwomendifferintheirqualityoflifeaftersurgeryforbreastcancerQuestion:Doyoungerandolderwomendifferintheirqualityoflifeaftersurgeryforbreastcancer?

Survey2:AnxietyinSchool

Objective:TodeterminethenatureandtypeofanxietyassociatedwithschoolQuestion:Doboysandgirlsdifferintheiranxieties?Howdoyoungerandolderstudentscompare?

Eachobjectiveorquestioncontainsindependentanddependentvariables.Independentvariablesareusedtopredictorexplainthedependentvariables.Theyoftenconsistofthegroups(experimentalorcontrol,menorwomen)towhichrespondentsbelongortheircharacteristics(under50yearsoldand51yearsofageandolder).Takethequestion,“Doboysandgirlsdifferintheiranxieties?”Thegroupingorindependentvariableissex.

Thedependentvariablesaretheattitudes,attributes,behaviors,andknowledgethesurveyismeasuring.Instatisticalterms,theyarethevariablesforwhichestimatesaretobemadeorinferencesdrawn.Inthequestion,“Doboysandgirlsdifferintheiranxieties?”thedependentvariableisanxieties.

Identifysubgroups.

Thesubgroupsrefertothegroupswhosesurveyresultsmustbeobtainedinsufficientnumbersforaccurateconclusions.Inthetwoprevioussurveys,thesubgroupscanbeidentifiedbylookingattheindependentvariables.Survey1’ssubgroupsareolderandyoungerwomen.Survey2’ssubgroupsareolderandyoungerboysandgirls.

Identifysurveytypeanddatacollectionneeds.

Thedependentvariablestellyouthecontentofthesurvey.Forexample,Survey1’sspecificquestionswillaskrespondentsaboutvariousaspectsoftheirqualityoflife.Survey2’swillaskaboutanxiety.SupposeSurvey1isaface-to-faceinterviewandSurvey2isaself-administeredquestionnaire.

Checkthesurvey’sresourcesandschedule.

Asurveywithmanysubgroupsandmeasureswillbemorecomplexandcostlythanthosewithfewsubgroups.Considerthis:

Thenumberofsubgroupsrangesfromtwotofour.Administering,scoring,andinterpretingthesurveyforonegroupisdifficultenough;withmorethanone,thedifficultiesmount.

Calculatesamplesize.

Supposeasurveyisconcernedwithfindingoutwhetheraflexible-timeworkprogramimprovesemployeesatisfaction.Supposealsothatonesurveyobjectiveistocomparethegoalsandaspirationsofemployeesintheprogramwithothernonparticipatingemployees.Howlargeshouldeachgroupofworkersbe?Toanswerthisquestion,fiveotherquestionsmustbeanswered.

FiveQuestionstoAskWhenDeterminingSampleSize1.Whatisthenullhypothesis?Thenullhypothesis(Ho)isastatementthatnodifferenceexistsbetweentheaverageormeanscoresoftwogroups.Forexample,onenullhypothesisforthesurveyofemployeesatisfactionisthis:

Ho=Nodifferenceexistsbetweengoalsandsatisfaction(asmeasuredbyaveragesurveyscores)betweenemployeesparticipatingintheprogramandnonparticipatingemployees.

2.Whatisthedesiredlevelofsignificance(alevel)relatedtothenullhypothesisinvolvingthemeaninthepopulation(mo)?Thelevelofsignificance,whenchosenbeforethetestisperformed,iscalledthealphavalue(denotedbytheGreekletteralpha:α).Thealphagivestheprobabilityofrejectingthenullhypothesiswhenitisactuallytrue.Traditionkeepsthealphavaluesmall—.05,.01,or.001—toavoidrejectinganullhypothesiswhenitistrue(andnodifferenceexistsbetweengroupmeans).Thepvalueistheprobabilitythatanobservedresult(orresultofastatisticaltest)isduetochance(ratherthantoparticipationinaprogram).Itiscalculatedafterthestatisticaltest.Ifthepvalueislessthanalpha,thenthenullisrejected.

Whendifferencesarefoundtoexistbetweentwogroups,butinrealitytherearenodifferences,thatis

calledanalphaorTypeIerror.Whennodifferencesarefoundbetweengroups,althoughinrealitythereisadifference,thatistermedabetaorTypeIIerror.

3.Whatchanceshouldtherebeofdetectinganactualdifference?Poweristheabilitytodetectadifferenceofagivensizeifthedifferencereallyexists.Itiscalculatedas1–β(Greekletterbeta).Itisdefinedastheprobabilityofrejectingthenullhypothesiswhenitisfalseoracceptingthealternativehypothesiswhenitistrue.Youwanthighpower.

4.Whatdifferencesbetweenmeansareimportant?Thatis,whatisameaningfulm1–m2?SupposethesurveyusestheGoalsandSatisfactionScale(GASS).Thishypotheticalscalehas50points.Ifthescaleisvalid,youwillhaveaccesstopublishedscoringrulesanddatadescribingwhatthescoresmean.Askthesequestions:Arehigherscoresbetter?Howmany“points”makeapractical(educationalorclinical)difference?Theanswerstoquestionssuchasthesewillhelpyoudecidehowmuchdifferenceyouwanttodetectinyourtwogroups.

5.Whatisagoodestimateofthestandarddeviationsinthepopulation?Thestandarddeviation(s,lowercaseGreeklettersigma)isacommonmeasureofdispersionorspreadofdataaboutthemean.Ifthedistributionofvaluesorobservationsisabell-shapedornormaldistribution,then68%oftheobservationswillfallbetweenthemean±1standarddeviation;95%oftheobservationsbetween±2standarddeviations;and99%oftheobservationsbetween±3standarddeviations.Lookatthis:

Estimatesofthestandarddeviationcancomefrompreviouslydonesurveys.Checkthatthesampleusedtoderivethestandarddeviationissimilartoyourown.Ifitisnot,thestandarddeviationinyourgroupislikelytobedifferentandsoisyourgroup.

Anotherwaytoestimatethestandarddeviationinyourgroupistoconductasmallpilottestusingabout25to50people.Youcanalsohaveexpertsgiveyouestimatesonthehighestandlowestvaluesorscoresasthebasisforcalculatingthestandarddeviation.

Whatfollowsisoneformulaforcalculatingsamplesizeforcomparingthemeansfromtwoindependentgroups(suchastwogroupsofemployees).Group1isinaprogramtoimprovesatisfaction,butGroup2isnot.Thisformulaassumesthatthetwogroups’samplesizesandstandarddeviationsareequal:(zα−zβ)σ2μ1−μ2

where

μ1–μ2isthemagnitudeofthedifferencetobedetectedbetweenthetwogroupszαtheuppertailinthenormaldistribution)andzβ(thelowertail)aredefinedas

zαX–μ1σ/nandzβX–μ2σ/n

Hereisanexampleofhowtoapplytheformula.

Example:CalculatingSampleSizeinaSurveyofEmployeesinanExperimentalandControlGroupSurveysituation.Devsoft’semployeesareparticipatinginaprogramtoimprovetheirjobsatisfaction.Attheconclusionofthe3-yearprogram,participantsintheexperimentalandcontrolgroupswillbesurveyedtofindoutabouttheirgoalsandaspirations.Thehighestpossiblescoreonthesurveyis100points.TheType1oralphalevelissetat.05.Theprobabilityofdetectingatruedifferenceissetat.80.Apanelofexpertsinsatisfactionmeasuressaysthatthedifferenceinscoresbetweentheexperimentalandcontrolgroupsshouldbe10pointsormore.Previousemployeesurveysfoundastandarddeviationof15points.

Thecalculations.Forthecalculation,assumethatastandardnormaldistributionorzdistributionisappropriate.Thestandardnormalcurvehasameanof0andastandarddeviationof1.(Formoreaboutthestandardnormaldistribution,one-andtwo-tailedtests,andzvalues,seeChapter6.Actualzvaluesareobtainableinstatisticsbooksandprogrammanuals.)Thetwo-tailedzvaluerelatedtoα=.05is+1.96.Forα=.01,thetwo-tailedzvalueis2.58;forα=.10,thezvalueis1.65;andforα=.20,thezvalueis1.28.Thelowerone-tailedzvaluerelatedtoβis-.84(thecriticalvalueorzscoreseparatingthelower20%ofthezdistributionfrom80%).Applyingtheformula,wehavethefollowing:(1.96+0.84)(15)2=2(4210)2=2(17.64)orabout36

Atleast36employeesareneededineachgrouptohavean80%chanceofdetectingadifferenceinscoresof10points.

Sometimes—forpracticalreasons—youcanassembleonlyacertainnumberofpersonsforyoursurvey.Howdoyouknowwhetherthenumberislargeenoughtofinddifferences?Again,statisticalmethodstotherescue.Lookatthis.

Example:PowertoDetectDifferencesTheAlcoholRehabilitationUnithasaprogramtoreducerisksfromalcoholuse.Itsetsthisstandardofsuccess:

Attheendoftheprogram,20%oftheharmfuldrinkersinthetreatmentgroupwillreducetheirrisks,whereas10%oftheharmfuldrinkersinthecontrolgroupwilldoso.

Theunithopestobeabletosurvey150personsineachgroupregardingtheirrisks.Isthissamplelargeenough?Astatisticianiscalledintoanswerthequestion.Thestatisticianproducesthistableforcomparingpercentagechangesusingatwo-sidedtestwithα=.05.

Youinterpretthistableasfollows:

Ifwehave150persons(finalanalyticsamplesize)intheexperimentalgroupand150inthecontrolgroup,wewillhave69%powertodistinguishashiftof20%intheexperimentalgroupfromharmfultolessriskyfromashiftof10%inthecontrolgroup.

Youcanuseastatisticalapproachtodistinguishdifferenteffectsizes.Theeffectsizeisthedifferencedividedbythestandarddeviation.Lookatthis.

Example:80%PowerandEffectTheAlcoholRehabilitationUnitwillhavecompletesurveydataon150personsinanexperimentalgroupand150inacontrolgroup.Thesepersonswillhavecompletedasurveybeforetheyparticipateintheunitandimmediatelyafter.Howmuchpowerwillthissamplesizeyield?Poweristheabilitytodetectadifferenceoreffect.Putanotherway,itistheabilityofastatisticaltesttodetectanalternativehypothesisofdifference(betweengroups)ofaspecifiedsizewhenthealternativeistrue(andthenullisrejected).Astatisticianprovidesthistable.Whatdoesitmean?

Youinterpretthistableasfollows:

Ifwehave150(finalanalyticsamplesize)inourexperimentalgroupand150inourcontrolgroup,wewillbeabletodistinguisha.32effectsizebetweenthedifferenceovertimeintheexperimentalgroupversusthedifferenceovertimeinthecontrolgroup.

Becauseyouestimateeffectbydividingthemeanoraveragedifferenceinscoresbythestandarddeviation,youcansee,forexample,whethertheexperimentalgroup’smeanimprovementhadastandarddeviationof.50andthecontrolgroup’smeanimprovementhadoneof.10.Ifso,thentheeffectsizewouldbe50-10=40>.32.Inthiscase,youwillhaveatleast80%powertodetectthedifferenceof.40betweenthetreatmentandcontrolgroups.

Theprecedingdiscussionistohelpyoulearnterminologyandtoaidyouinevaluatingtheusefulnessofyoursamplingmethodsandoutcomes.Samplesizes,effects,andpowerarestatisticalterms.Unlessyouplantolearnstatistics,callinanexperttohelp.Alternatively,considerusingthesample-sizecalculatorsfoundinstatisticalprogramsorontheweb.Gotoyourfavoritesearchengineandtypein“samplesize”tofindfreesample-sizecalculators.

Thefollowingillustrateshowtowriteupandjustifysamplesize,effect,andpower.

Example:SampleSize,Effect,andPower

1.YogaforChronicBackPainTheUnitedKingdomBEAM(BackPainExerciseandManipulation)trialfoundthatachangeintheRolandMorrisDisabilityQuestionnairescoreof1.57pointswasacost-effectivedifference.Assumingastandarddeviationof4points(asintheUnitedKingdomBEAMsamplesize),thisresultsinaneffectsizeof0.39[1.57÷4].Todetectthiseffectsize,assuming80%powerand20%attrition,werequiredatotalof262participants(131pergroup).

Ifyouwouldliketolearnmoreaboutthisstudy,gotoTilbrook,H.E.,Cox,H.,Hewitt,C.E.,Kang’ombe,A.R.,Chuang,L.-H.,Jayakody,S.,etal.(2011).Yogaforchroniclowbackpain.AnnalsofInternalMedicine,155(9),569–578.

2.EffectivenessofTelephoneCounselingforSmokingCessationinParentsBasedonsimilarstudies,weexpecteda6%differencein7-daypointprevalenceabstinenceratesbetweenthetelephonecounselingconditionandthecontrolconditionat12-monthsassessment(13%versus7%,respectively).Astatisticalpowerof.80wastargeted.Hypotheseswillbetestedatatwo-sidedsignificancelevelof.05.Thecalculatedsamplesizewascorrectedforparticipantswhowillbelosttoattrition.Additionally,thesamplesizewascorrectedtoallowforsupplementaryanalysesofmediationandmoderation.

Ifyouwouldliketolearnmoreaboutthisstudy,gotoSchuck,K.,Otten,R.,Kleinjan,M.,Bricker,J.,&Engels,R.(2011).Effectivenessofproactivetelephonecounsellingforsmokingcessationinparents:Studyprotocolofarandomizedcontrolledtrial.BMCPublicHealth,11(1),732.

ResponseRateTheresponserateisthenumberofpersonswhorespond(numerator)dividedbythenumberofeligiblerespondents(denominator).If100peopleareeligibleand75completedsurveysareavailableforanalysis,theresponserateis75%.

Allsurveyshopeforahighresponserate.Nosinglerateisconsideredthestandard,however.Insomesurveys,between95%and100%isexpected;inothers,70%isadequate.

Herearesometipstoimprovetheresponserate.

TipsforImprovingResponseRateKnowyourrespondents.Makecertainthequestionsareunderstandabletothem,tothepoint,andnotinsensitivetotheirsocioculturalvalues.Usetrainedpersonneltorecruitrespondentsandconductsurveys.Setupaqualityassurancesystemformonitoringqualityandretraining.Identifyalargernumberofeligiblerespondentsthanyouneedincaseyoudonotgetthesamplesizedesired.Becarefultopayattentiontothecosts.Usesurveysonlywhenyouarefairlycertainthatrespondentsareinterestedinthetopic.Keepsurveyresponsesconfidentialoranonymous.Sendreminderstononresponders.Provideoptionsforcompletingthesurvey.Forinstance,somepeoplepreferonlinesurveys,whereasotherspreferpaper-and-pencilsurveysthattheycompleteandreturnbymailorfax.Providegiftorcashincentives.Berealisticabouttheeligibilitycriteria.Anticipateinadvancetheproportionofrespondentswhomaynotbeabletoparticipatebecauseofsurveycircumstances(suchasincorrectaddresses)orbychance(theysuddenlygetill).Tellrespondentshowyouwillusethesurvey’sdata.Bepreciseindescribinghowprivacyissafeguarded.

WeightingSurveysrarelyobtaininformationfromeveryonewithinthepopulationofinterest.Tomakeupforthediscrepancy,andtoensurethatthesurveyresultsarerepresentativeofthepopulation,surveyorsoftenassignweightsorvaluestoeachcase.Supposeyousurvey100peoplewhoarebetween15to64yearsofageandyouobtainresponsesfrom60femalesand40males.Forthatagerange,youshouldhaveresponsesfrom50malesand50females.Tocorrectforthisdifference,youassignaweighttoeachmale’sanswerssothattheycontributemore(moreweight)thandoeachfemale’s.

Thetwomostcommontypesofweightsaredesignweightsandpoststratificationornonresponseweights.Designweightsareusedtocompensateforoversamplingorundersamplingofspecificcasesorfordisproportionatestratification.Sayyouwanttocompare50membersofGroupAand50membersofGroupB.YoulearnthatGroupAhasmanypeoplewhooftenrefusetotakesurveys.YoudecidetosamplemorepeopleinGroupA(oversample)thaninGroupB,hopingthatevenwitharelativelylargerefusalrate,youwillendupwithequalnumbersofcompletedsurveysinbothgroups.IfyouoversampledGroupAatarate4timesgreaterthantherateforGroupB,thanthedesignweightforGroupAis¼,andforGroupB,itis1.

Poststratificationornonresponseweightsarealsousedtocompensateforthefactthatpersonswithcertaincharacteristicsarenotequallyaslikelytorespond.Mostgeneral-populationsurveyshavesubstantiallymorefemalethanmalerespondents(often60/40),althoughthereareoftenmoremalesinthepopulation.Becausethesurveywilloverrepresentfemalesandunderrepresentmales,youassignaweighttomakeupforthisbias.

Therearemanyrespondentcharacteristicsthatarelikelytoberelatedtothetendencytorespond,includingage,education,race/ethnicity,sex,andplaceofresidence.Itisusuallymoredifficulttocomeupwithpoststratificationthandesignweightsbecauseyouneedsupplementaryinformationaboutthepopulation.Thismeansidentifyingandanalyzingverylargedatabases(suchasacountry’scensusorauniversity’senrollmentrecords)togetestimatesofthedemographicvariablesthatinterestyou.Thesedatabasesprovideanswerstoquestionslikethefollowing:Whatistheactualproportionofmenandwomeninthepopulation?Whatistheprevalenceofthisdiseaseinyoungerversusolderpeople?Whatpercentageofpeopleliveinsingle-familyhomescomparedtootherlivingarrangements?

Example:CalculatingPoststratificationWeightsHereisahypotheticalexampleofhowpoststratificationweightsarecalculated:

MarginofErrorandConfidenceLevelIfyoureadsurveyreports,youmayhavenoticedastatementlike:Samplingerrorforthetotalsampleof1,066respondentsisplusorminus3.2percentagepointsatthe95%levelofconfidence.Samplesizesandsamplingerrorsforothersubgroupsareavailableuponrequest.

Marginoferrorandconfidencelevelaretwotermstypicallyassociatedwithpolls.Apollisasurveyofpeople’sopinionsataparticularpointintime.Mostpeoplearefamiliarwithpoliticalpollsinwhichthepublicisaskedaboutapoliticalparty’sorcandidate’sperformanceandwhotheyintendtovotefor.Marginoferrorandconfidencelevelareimportantinpollsbecauseofthesurveyor’sconcernthatthesampleofrespondentsmaynotaccuratelyrepresentthepopulation.Forinstance,ifyoupoll1,000studentstofindoutwhethertheyplantovoteinthiselectionforStudentBodyPresidentand700(70%)saytheyplantovote,youneedtobeconfidentthatthe70%islikelytorepresentthe1,000.Youwantthemtobesimilartothe1,000inage,experience,andmotivation.

Technically,themarginoferrorisastatisticexpressingtheamountofrandomsamplingerrorinasurvey’sresults.Thelargerthemarginoferror,thelessconfidenceonehasthatthepoll’sreportedresultsareclosetothe“true”figures,whicharethefiguresforthewholepopulation.

Howwellthesamplerepresentsthepopulationisgaugedbytwoimportantstatistics:thesurvey’smarginoferrorandtheconfidencelevel.Asurveymayhaveamarginoferrorofplusorminus3%ata95%levelofconfidence.Thismeansthatifthesurveyisconducted100times,thedatawillbewithin3percentagepointsaboveorbelowthepercentagereportedin95ofthe100surveys.

Supposeyouwanttofindoutifparentsaresatisfiedwithopportunitiesforinvolvementinschoolactivities.Yousurveyparentsandfindthat50%are“definitelysatisfied.”Youmightthenreportthattheresultsareaccurateplusorminus3percentagepointsatthe95%confidencelevel.Youcouldalsostatetheresultsthisway:“IfIweretoconductthesurvey100times,thepercentageofparentswhosaytheyaresatisfiedwithopportunitiesforinvolvementwillrangebetween47and53%95%ofthetime.”

Assumingyouwantaconfidencelevelof95%,considerthis:

SampleSizeandtheMarginofErrorLookingatthischart,youcanseethatthemarginoferrordecreasesasthesamplesizeincreases,butonlytoapoint.Averysmallsample,suchas50respondents,hasabouta14%marginoferrorwhileasampleof1,000hasamarginoferrorof3%.Thesizeofthepopulation(thelargergroupfromwhichthesampleistaken)doesnotmatter.Thereisapointofdiminishingvalue,however.Bydoublingthesampleto2,000,themarginoferroronlydecreasesfromplusorminus3%toplusorminus2%!

Althougha95%levelofconfidenceisastandardinbusiness,health,politics,andeducation,a90%levelmaysufficeinsomeinstances.A90%levelcanbeobtainedwithasmallersample,whichusuallytranslatesintoalessexpensivesurvey.Toobtaina3%marginoferrorata90%levelofconfidencerequiresasamplesizeofabout750.Fora95%levelofconfidence,theneededsamplesizeisabout1,000.

Determiningthemarginoferroratvariouslevelsofconfidenceisrelativelyeasy.Gotothewebforthemarginoferrorforvariousconfidencelevels.Youcanask:Whatisthemarginoferrorfora90%(95%)confidencelevel?Youcanalsogotothewebtofindsitesforcalculatingthesamplesizeneededtoobtainaspecificmarginoferror.Ifthesurveyorisinterestedinamarginoferrorof3%,forinstance,thecalculatorwillsaywhatthesamplesizeshouldbe.Gotothewebandask:Whatisthesamplesizeneededfora90%(95%)confidencelevelanda3%marginoferror?

SummingUpThreeofthemostcommonlyusedrandomsamplingmethodsarethefollowing:1. Simplerandomsampling2. Stratifiedrandomsampling3. Simplerandomclustersampling

Randomsamplinggiveseveryonewhoiseligibletoparticipateinthesurveyafairchanceofselection.Youcansampleindividualsorlargerunitssuchasschools,offices,andhospitals.Theselargersamplingunitscontainclusters(ofstudents,employees,nurses,physicians,patients),sothetechniqueiscalledclustersampling.Nonrandomsamplesareofteneasiertoassemblethanrandomsamples.Butgainsineasecanbemetwithlossesingeneralizability.

Withsystematicsamples,everynth(e.g.,every5thor500th)unit(individuals,schools,factories)isselectedfromalistofunits.Ifnisrandomlyselected,systematicsamplingbecomeslikerandomsampling.Inconveniencesampling,youselecteveryonewhoisavailablewhenyouneedthemiftheymeetthecriteriaforyoursurvey(rightageorreadinglevel,votedinthelastelection,havelivedinthecommunityforatleast1year,etc.).Othernonprobabilitysamplingmethodsincludesnowballandquotasampling.Othernonrandomsamplingstrategiesincludethefollowing:

Snowballsamples.Previouslyidentifiedmembersofagroupidentifyothermembers.Forexample,youselectCEOsandaskthemtonominateothers.Quotasamples.Thegroupisdividedintosubgroupsinspecificproportions.Thisissimilartostratifiedsampling.Focusgroups.Tento20peoplearebroughttogethertoanswerspecificquestions.Atrainedleaderconductsthesessions.Atranscriberisusuallypresenttosummarizethediscussion.Panels.Respondentsagreetocompletesurveysiftheyarecontacted.Therespondentsareprescreenedfortheirdemographicandothercharacteristics(e.g.,frequentInternetshoppers,peoplewhoneveruseTwitter).

Largersamplestendtoreducesamplingerrorswhenthesamplesarerandomlyselected.Thestatisticusedtodescribesamplingerroriscalledthestandarderrorofthemean.Itisthestandarddeviationofthedistributionofsampleestimatesofmeansthatcouldbeformedifaninfinitenumberofsamplesofagivensizeweredrawn.Surveysrarelyobtaininformationfromeveryonewithinthepopulationofinterest.Tomakeupforloss,andtoensurethatthesurveyresultsarerepresentativeofthepopulation,surveyorsassignweightsorvaluestoeachcase.Therearetwotypes:designandpoststratification,ornonresponse,weights.Themarginoferrorisastatisticexpressingtheamountofrandomsamplingerrorinasurvey’sresults.Thelargerthemarginoferror,thelessconfidenceyoushouldhavethatthepoll’sreportedresultsareclosetothe“true”figures,whicharethefiguresforthewholepopulation.

ThinkAboutThis1. Locatethesurveysinthearticlesbelowandthendothefollowing:

1. Discussthetypeofsamplingthatthesurveyorsuse.2. Describethesurvey’seligibilitycriteriabylistingthecharacteristicsofrespondentswhoare

includedandexcluded.2. Findanythreesurveysthatarepartoflargerstudiesanddiscusstheirsamplingplans.Arethe

samplingplansdiscussedinadequatedetail?Thatis,doyouthinkyouhaveenoughinformationaboutthesamplingstrategyandsamplingsizetoredothesurveyifnecessary?Whatadditionalinformationwouldyouneed?

3. Listatleastfivetipsforimprovingasurvey’sresponserate.4. Explainthefollowingtermsastheyareusedwhentalkingaboutsurveys:standarderror,marginoferror,confidencelevel,confidenceinterval,power,effectsize,levelofsignificance,andnormaldistribution.

5. InSeptember2014,thePewResearchInternetProjectsponsoredasurveytostudytechnology’seffectsontheAmericanworkplace.Gotothesite(http://www.pewinternet.org/2014/12/30/technologys-impact-on-workers/?

http://www.pewinternet.org/2014/12/30/technologys-impact-on-workers/?utm_source=Pew+Research+Center&utm_campaign=5db76b36bf-Work_Productivity_123014&utm_medium=email&utm_term=0_3e953b9b70-5db76b36bf-399429165

utm_source=Pew+Research+Center&utm_campaign=5db76b36bf-Work_Productivity_123014&utm_medium=email&utm_term=0_3e953b9b70-5db76b36bf-399429165)andanswerthesequestions:1. Howwasthesurveyadministered?2. Whichsamplingmethodwasused?3. Whatweretheeligibilityrequirements?4. Describethefinalsampleintermsofitssizeanddemographics.5. Whoconductedthesurvey?Whataretheirqualifications?6. Whatisthemarginoferror(samplingerror)?Whatisthelevelofconfidence?7. Describethesurvey’smainfindings.

ReferencesHolman,E.A.,Silver,R.C.,Poulin,M.,Andersen,J.,Gil-Rivas,V.,&McIntosh,D.N.(2008).Terrorism,acutestress,andcardiovascularhealth:A3-yearnationalstudyfollowingtheSeptember11thattacks.ArchivesofGeneralPsychiatry,65(1),73–80.

O’Connor,T.M.,Yang,S.-J.,&Nicklas,T.A.(2006).Beverageintakeamongpreschoolchildrenanditseffectonweightstatus.Pediatrics,118(4),2005–2348.

Schuck,K.,Otten,R.,Kleinjan,M.,Bricker,J.,&Engels,R.(2011).Effectivenessofproactivetelephonecounsellingforsmokingcessationinparents:Studyprotocolofarandomizedcontrolledtrial.BMCPublicHealth,11(1),732.

Skoffer,B.(2007).Lowbackpainin15-to16-year-oldchildreninrelationtoschoolfurnitureandcarryingoftheschoolbag.Spine,32(24),E713–E717.

Tilbrook,H.E.,Cox,H.,Hewitt,C.E.,Kang’ombe,A.R.,Chuang,L.-H.,Jayakody,S.,etal.(2011).Yogaforchroniclowbackpain.AnnalsofInternalMedicine,155(9),569–578.

Wehr-Flowers,E.(2006).Differencesbetweenmaleandfemalestudents’confidence,anxiety,andattitudetowardlearningjazzimprovisation.JournalofResearchinMusicEducation,54(4),337–349.

5SURVEYDESIGNEnvironmentalControl

OverviewAsurveycanbegiventojustonegrouportomanygroupsofpeopleonceorseveraltimes.Forexample,youcansurveypreschoolchildreninoneschooljustbeforeandimmediatelyaftertheyparticipateinanewmusicprogram,oryoucansurveypreschoolchildrenin10schoolsjustbeforetheyparticipateintheprogramandevery2yearsafteruntiltheycompletehighschool.Asurvey’sdesignreferstoitsfrequency(onetimeormoreoften),sequence(justbeforeandimmediatelyafteranewprogram),thenumberofgroupsinvolved(onegroupconsistingofallstudentsinaschooldistrictortwoormoregroups,oneofwhichispartofanexperimentornewprogram),andhowthegroupswereassigned(atrandomornotatrandom).

Somedesignsareobservationalandothersareexperimental.Across-sectionalsurveydesignprovidesaportraitorobservationofthingsastheyareatasinglepointintime.Apollofvoters’preferences1monthbeforeanelectionandasurveyoftheincomeandageofpeoplewhovoteinaparticularelectionbothusecross-sectionaldesigns.

Longitudinalsurveysareobservationaldesignsthatareusedtofindoutaboutchange.Ifyoutakeasampleofchildrenwhoweresixthgradersin2015andfollowthemwithasurveyeveryyearforthenext5years,youareusingalongitudinaldesign.Othersurveydesignsinvolvecomparisonsbetweenoramonggroups.Theseareexperimentaldesigns.Ifrespondentsareassignedtogroupsonarandombasis,andeveryrespondenthasanequalchanceofassignmenttoagroup,youhavearandomizedcontrolledtrialortrueexperiment.Nonrandomizedcontrolledtrialsorquasi-experimentaldesignsarecomparisongroupdesignsinwhichtherespondentsareassignedtogroupsinanon-randomway.

Validsurveyfindingscomefromvalidsurveys(measurementvalidity)andvalidstudydesigns(internalandexternalvalidity;minimalbias).Validdesignsaredependentonfactorssuchastheobjectivityofthesurveyparticipants’selection,thestabilityovertimeoftheirresponses,therespondents’psychologicalandphysicalgrowth,andeventsintheoutsideworld.Thesefactors,ifnotunderstoodandaddressed,threatentheinternalvalidityofthesurvey’sdesignandbiastheresults.Whenthefactorsaffecttheapplicationofthefindingstoothersettingsandotherrespondents,thentheexternalvalidityorgeneralizabilityofthesurvey’sfindingsisthreatened.

WhichDesignsAreAvailable?Surveysmaytakeplaceontheirown.Acompanypollofitsemployees’satisfactionisanexampleofastand-alonesurvey.Butsurveysareoftenincludedaspartofalargerstudy,andthatstudymayincludeotherdatacollectionmethodsaswell.Supposeacompanyisinterestedinimprovingemployeeperformance.Thecompanyinstitutessomenewprogramsandrandomlyassignssomeemployeestothemwhiletheotherscontinueasusual.After6months,thecompanyinterviewsbothgroupsofemployeestofindoutabouttheirsatisfactionwithbothprograms,observesthematwork,andreviewstheiron-the-jobperformance.Onlyoneofthemeasures—theinterviews—isactuallyasurvey.Nevertheless,thesurveyorneedstobesurethatthedesignofthestudytoevaluatethenewprogramsdoesjusticetotheneedsofthesurvey.

Surveydatacanbeusedtodescribethestatusofthings,showchange,andmakecomparisons.Thesurvey’sdesignreferstothewayinwhichthe“environment”iscontrolledororganized.Theenvironmentreferstothenumberandcharacteristicsofrespondentsandhowoftenandwhentheywillbesurveyed.Themoreenvironmentalcontrolyouhave,themoreaccurateyourresultswillbe.

Theenvironmentalvariablesoverwhichsurveyorshavethemostcontrolareasfollows:(a)whenthesurveyistobegiven(e.g.,immediatelyaftergraduation),(b)howoften(e.g.,twice:immediatelyaftergraduationand12monthsafterthat),(c)thesamplesize(e.g.,10or1,000graduates),(d)thenumberof

groups(e.g.,justone,allgraduates;ormorethanone,graduatesofpublicschoolsandgraduatesofcharterschools),and(e)howthegroupswereassigned(atrandomornotrandomly).

LookatthesesurveysplannedbytheHave-A-HeartAssociation.

Example:SurveysWithDifferingDesigns1. TheHave-A-HeartAssociationoffers10educationalprogramstopeopleinthecommunity.InJune,itisconductinga

surveyofarandomsampleofpeopletofindoutanddescribewhichofthe10programstheyselectandwhy.2. TheHave-A-HeartAssociationwantstoknowwhetherpeoplewhoparticipateinitseducationalprogramsgainknowledge

abouthowtolessentheirriskofaheartattack.Theassociationconductedsurveyswith300volunteerparticipantsfromtheclassof2015.Thesurveyswereconductedonceayearfor3yearsandtheresultswerecomparedfromtimetotimetoseewhetherknowledgegainsendure.

3. TheHave-A-HeartAssociationhasconductedanevaluationoftheeffectivenessoftwocompetingprogramsthataimtoreducerisksforheartdiseasethrougheducation.Participantswereselectedatrandomfromtheassociation’sdatabase.Theywerethenrandomlyassignedtoparticipateinoneofthetwoprograms.Participants’risksforheartdiseaseweresurveyedbeforeprogramparticipationand6and12monthsafter.

ThefirstHave-A-HeartAssociationsurveyistobeconductedinJunetodescribetheprogramsthatthecommunityselects.Thedesign,inwhichdataarecollectedatonepointintime,iscalledcross-sectional.

ThesecondHave-A-HeartAssociationsurveycallsforcollectinginformationovera3-yearperiodfromasingleclass.Thisisalongitudinal,time-series,orcohortdesign.

ThethirdHave-A-HeartAssociationsurveyisusedaspartofanevaluationoftheeffectivenessoftwoprograms.Itusesacomparisongroupdesign.Thedesignisexperimental.

Table5.1showstherelationshipamongthepurposes,samplinganddesignconcerns,results,andtypeofdesignforthethreeHave-A-HeartAssociationsurveys.

Cross-SectionalSurveyDesignsWiththisdesign,dataarecollectedatasinglepointintime.Thinkofacross-sectionalsurveyasasnapshotofagroupofpeopleororganizations.

SupposetheHave-A-HeartAssociationwantstoknowwhichofitseducationalprogramsthecommunityprefers.Considerthisquestionanditsanswer:

Example:Cross-SectionalDesignQuestion:Ifonlyoneprogramwerepossible,whichwouldyouchoose?Sample:Across-sectionof500people,randomlyselected,whoattendedaneducationprogramthisyearDesign:Cross-sectionalMethod:TelephoneinterviewsAnswer:DineOutwins.TheevidenceisseeninTables5.2and5.3.

Assumingthatasampleofparticipantshasbeenwiselychosenbyarandomsamplingtechniqueandtherightquestionshavebeenasked,thetablesintheexamplerevealthatDineOutisthewinner.Thisiswhy:

1. Regardlessofsexorage,DineOutisahead.2. MorementhanwomenpreferFeelFitandEmergencyCare.ButwhenitcomestoDineOut,menand

womenhavenearlythesamepreference.3. Peopleover65yearsofagepreferFeelFit,buttherearenotasmanyofthemaspeopleintheother

twocategories.

Ofcourse,youmightwanttouseonlyTable5.2orjustTable5.3tomakeapointaboutyoursurvey.Youcanalsocombinethemintoonelargetable.Butacarefullyplannedcross-sectionaldesignwillgiveyouavarietyofwaysforanalyzingandpresentingyoursurveydata.

Withthisprogrampreferencedata,forexample,youmightalsohaveconsideredprofession:Howdopeopleinbusinessandtheprofessionscompare?Doesretirementmakeadifference?Oryoumighthaveconsideredresidence:Howdopeopleinonepartofthecitycomparewithpeopleinsomeotherpart?

MakingtheDecisionCross-sectionalsurveyshaveimportantuses.Theydescribethingsastheyaresothatpeoplecanplanforchange.Iftheyareunhappywiththepicturethatacross-sectionalsurveyreveals,theycanchangeit.

Cross-sectionalsurveysarealsorelativelyeasytodo.Theyarelimited,however,inthatifthingschangerapidly,thesurveyinformationwillpossiblybecomeoutdated.

LongitudinalSurveysLongitudinalsurveydesignreferstothecollectionofsurveydataovertime.Theyaresometimescalledtime-seriesdesigns.Acohortdesignreferstosurveyingaparticulargroup(e.g.,sixthgraders)morethanjustonce(say,onceayearfor3years).LookatthisexamplewithparticipantsinaprogramsponsoredbytheHave-A-HeartAssociation.

Example:LongitudinalSurveys:CohortDesignQuestion:Whatdoparticipantsknowaboutheartdisease?Sample:Avolunteersampleof300participantswhoattendedDineOut,whichaimstoteachpeoplethecausesofheartdisease,goodnutrition,andhowtopreventheartdiseaseevenwhendiningoutMethod:Onlineandmailedself-administeredquestionnairesAnswer:Inall3yearsofthesurvey,participantsconsistentlyknowlittleaboutdiseaseprevention.By2009and2010,theyhavesomeknowledgeaboutnutritionand,by2011,theyknowmuchaboutthecausesofheartdisease.

Proofisdisplayedinthefollowingtable.

MakingtheDecisionCohortsprovidethesurveyorwithdataonwhethereffects(say,ofaprogramlikeDineOut)lastovertime.Theproblemwithcohortdesignsisthat,unlesstherespondentsaremotivated,theymaydropoutandtheremaindermaynotbetypicaloftheentiregroup.Also,thegroupmaybeatypicalinthatitiswillingandhasthetimetoparticipateintheprogramandcompletethesurveys.

ExperimentalSurveyDesignsWiththesedesigns,peoplearedividedintotwoormoregroupsandtheirsurveyresultsarecompared.Theclassicalcomparisongroupdesigncontrastsan“experimental”groupwitha“placebo”or“usualcare”group.

Experimentaldesignsinsomefields(education,psychology)aresometimesdividedintoquasi-andtrueexperimentaldesigns.Inquasi-experimentaldesigns,creationandassignmenttogroupsareusuallydeliberateandnotrandom.Intrueexperimentaldesigns(randomizedcontrolledtrials),anindividual’sassignmenttogroupsisachanceresult.Trueexperimentsarethemorepowerfulofthetwobecause,ifproperlyimplemented,youcanconfidentlysayProgram1causedOutcome1.

Example:ANonrandomizedControlledTrialorQuasi-ExperimentalDesignYouhavebeenaskedtoevaluatetwoprogramsfortheelderly.Eligibleparticipantsareassignedtotheprogramsonafirst-come,first-servedbasis,resultingin60peopleinProgram1and59inProgram2.Oneoftheissuesaddressedbythesurveyiswhethertheparticipantsaresatisfiedwiththestaffoftheirprogram.Toanswerthisquestion,youaskparticipantsinbothgroupstocompleteaquestionnaireattheendof3monthsoftreatment.Thedesignforthissurveylookslikethis:

Howvalidisthequasi-experimentalcomparisongroupdesignusedforthesurveyoftheattitudesoftheelderlyintwoprogramstowardtheirstaff?Considerthesepossibilities:

1. Participantsinthetwogroupsmaybedifferentfromoneanotheratthebeginningoftheprogram.Forexample,olderpersonsmaychooseoneprogramovertheother.

2. Participantswhotrulydislikethestaffinoneoftheprogramsmayhavedroppedoutoftheprograms.

Example:ANonrandomizedControlledTrialWithaLongitudinalDesignAnotherquestionposedfortheevaluationofthetwoprogramsfortheelderlyiswhetherparticipantshavelearnedaboutcommonlyprescribeddrugs.Toanswerthisquestion,participantshavebeeninterviewedatthreetimes:atthebeginning,attheendofthefirstmonth,andattheendofthefirstyear.Thissurveydesignstrategycanbedepictedasfollows:

Thevalidityofthisdesignmaybethreatenedifpersonswithserioushealthproblemsarebychancemoreoftenassignedtooneprogramovertheotherorbyadifferentdropoutratefromonegroupcomparedwiththeother.

Example:ARandomizedControlledTrialorTrueExperiment

Thegovernmentcommissionedasurveytodeterminewhichofthreeprogramsfortheelderlywasthemosteffective.Amongtheconcernswasthecostoftheprograms.AcomparisongroupdesignwasusedinwhichpeopleattheFredaSmithCenterwererandomlyassignedtooneofthreeprogramsandtheaveragecostsofthethreeprogramsovera12-monthperiodwerecompared.

Thisdesignisrelativelystrong.Becausepeoplewererandomlyassignedtoeachprogram,anysourcesofchangethatmightcompetewiththeprogram’simpactwouldaffectallthreegroupsequally.However,rememberthat,althoughpeoplewereassignedrandomlytotheprogramswithintheFredaSmithCenter,othercentersmaydifferandthereforethefindingsfromthesurveymaynotbeapplicabletootherplaces.

Thistrueexperimentalandlongitudinaldesignisamongthemostsophisticatedandwillenableyoutomakesoundinferences.

Example:ARandomizedControlledTrialWithaLongitudinalDesignPrograms1and2intheFredaSmithCenterprovedtobeequallycost-effective.Thegovernmentthencommissionedastudytodeterminewhichprogramwasconsideredbyparticipantstodeliverbettermedicalcare.Tomakethedetermination,acomparisongroupdesignwasselectedinwhichcarewasassessed(usingascaleof10=excellentcare;1=poorcare)fromthebeginningto

theendofthe5-yearprogramandcomparedamongpeopleinPrograms1and2.The(hypothetical)surveythatwasusediscalledTheABYQualityofCareReviewSystemanditisasurveyofdoctors,nurses,andpatients.Thedesignisdepictedbythefollowingdiagram.

FactorialDesigns:SpecialCasesFactorialdesignsenableresearcherstoevaluatedifferentmethodsforofferinganewprogram,treatment,orpracticetoseewhichcombinationworksbest.Supposeyouareworkingonastudytofindoutiftheresponseratetoonlinesurveyscanbeimprovedbynotifyingprospectiverespondersinadvancebyemailand/orpleadingwiththemtorespond.Yourecommendusingatwo-by-two(2×2)factorialdesigninwhichsomerespondentsarenotifiedaboutthesurveyinadvancebyemail,whilesomearenot.Also,somepeoplearepleadedwithtorespond,whileothersarenot.Thefactors(theyarealsoindependentvariables)arepleading(Factor1)andnotifying(Factor2).Eachfactorhastwolevels:pleadversusdon’tpleadandnotifyinadvanceversusdon’tnotifyinadvance.

Ina2×2design,therearefourstudygroups:(1)prenotificationemailandpleadinginvitationemail,(2)prenotificationemailandnonpleadinginvitation,(3)noprenotificationemailandpleadinginvitation,and(4)noprenotificationandnonpleadinginvitation.Inthediagramabove,theemptycellsareplaceholdersforthenumberofrespondentsineachcategory(e.g.,thenumberinthegroupplead×notifyinadvancecomparedtothenumberinthegroupplead×don’tnotifyinadvance).Iftherespondentsarerandomlyassignedtoeachofthegroups,youwillhavearandomizedcontrolledtrialortrueexperiment.

Withthisdesign,theresearcherscanstudymaineffects(pleadversusdon’tplead)orinteractiveeffects(prenotificationandpleading).Theoutcomeinthisstudyisalwaystheresponserate.

Factorialdesignsmayincludemanyfactorsandmanylevels.Itisthenumberoflevelsthatdescribesthenameofthedesign.Forinstance,inastudyofpsychotherapyversusbehaviormodificationinoutpatient,inpatient,anddaytreatmentsettings,therearetwofactors(treatmentandsetting),withonefactorhavingtwolevels(psychotherapyversusbehaviormodification)andonehavingthreelevels(inpatient,daytreatment,andoutpatient).Thisdesignisa2×3factorialdesign.

Surveyparticipantsinfactorialdesignsmaybeselectedeitheratrandomorbyconvenience.

MakingtheDecisionRandomizedcontrolledtrialsarecharacterizedbyrandomselectionandassignment.Manyresearchersassertthatthebestrandomizedcontrolledtrialsarethoseinwhichparticipantsandinvestigatorsdonotknowtheirgroupassignment.Theyare“blinded.”Implementingrandomizedcontrolledtrialsmaybedifficultintherealworldofsurveysandsocialresearch.Theyarerelativelycostlyandrequiretimeforimplementationandanalysis.Nevertheless,ifyouwanttoknowwhetheranexperimenthasresults,therandomizedcontrolledtrialisthewaytogo.

Nonrandomizeddesignsareeasiertoimplement.However,becausegroupsarenotrandomlyconstituted,theirinitialdifferencesmayberesponsiblefororbecomeconfoundedwiththeoutcome.Thesurveyormustdoeverythingpossibletopreventtheseinitialdifferences,saybymatchingthegroupsonfactorsor

variablesthatmayinterferewithoutcomes.Thevariablesmayincludeeducation,motivation,income,andsoon.Statisticalmethodsareavailablethathelpcontrolthesedifferences.

OtherSurveyDesigns:NormativeandCaseControlTwolesser-knownsurveydesignsarethenormative(sometimescalledhistorical)andcasecontrol.Bothoffersomecontroloverthesurvey’senvironmentbymakinguseofspecialcomparisongroups.

NormativeSurveyDesignAnormisastandardforcomparinggroups.Normsthatarehistoricalcomefromexistingdataorlargedatabasessuchasthosemaintainedingovernmentregistries.Forinstance,normsareusedwhenaparticularschool’sreadingscoresarecomparedtothenationalaveragesfoundinacountry’seducationdatabase.Lookatthese.

Example:NormativeDesignParticipantsinLosAngeles’YouthProgramusedtheEFFICACYSurvey,aninstrumenttomeasureself-efficacythathadbeenvalidatedonanationalsampleof5,000people.ThenationalsamplewasusedasanormbecausetherewasnoreasontobelievethattheL.A.groupwoulddifferfromtherestofthenationinitsself-efficacy.YouthProgramscoreswerethuscomparedtothoseobtainedbythenationalsample.

Example:NormativeDesign—ComparisontoaModelArephysiciansinanewacademicmedicalcenterassatisfiedwiththeirworkasphysiciansinanolderone?Thenewcenterhasthesamemissionastheolderone.Toanswerthequestion,physiciansinbothcentersaresurveyedandthenewcenteriscomparedwiththemodeloldercenter.

MakingtheDecisionNormativesurveydesignscanbelessexpensiveandtime-consumingthanothercomparisondesignsbecausethesurveyordoesnothavetofindorcreateagrouptoactasthecontrol.Thesedesignsdependontheavailabilityofappropriatedatabases.Remember,yourgroupandthe“normal”onemayactuallydifferinimportantrespects,resultinginless-than-validfindings.SupposetheparticipantsintheYouthProgramofLosAngeleswereyoungerthanthenationalsample,forexample.Ifageandself-efficacywererelated,withmaturityassociatedwithbettersurveyscores,thentheYouthProgramwouldhavetoworkhardertobeasuccess.Ifyouusenormativedesigns,bepreparedtodefendyournormorstandardforcomparison.

CaseControlDesignAcasecontroldesignisoneinwhichgroupsofindividualsareselectedbecausetheyhave(thecase)ordonothave(thecontrol)theconditionbeingstudiedandthegroupsarecomparedwithrespecttoexistingorpasthealthcareconditions,attitudes,habits,beliefs,ordemographicfactorsjudgedtobeofrelevance

tothecausesofthecondition.

Casecontroldesignsaregenerallyusedbyresearcherswhoaretestingaspecifichypothesis—forexample,thataconnectionexistsbetweenlungcancerandcigarettesmokinghabits.

Acasecontroldesignneedstwogroups:acaseandthecontrol.Ideally,thetwogroupswilldifferonlyinthatthecasehasthecharacteristicsorconditionbeingstudiedandthecontroldoesnot.

Mostoftencasecontroldesignsmeanselectingacontrolthatislikethecaseinwaysthatarestronglysuspectedtoaffectthecondition.Ifyouhaveevidencethatgenderaffectsfrequencyofheadaches,thenastudyofpeoplewithheadachesshouldhavecasesandcontrolswithsimilarproportionsofmalesandfemales.Casecontroldesignsaremostoftenobservational.

MakingtheDecisionThemajorweaknessofthecasecontroldesignisthatthetwosurveygroupsmaynotbealikeatallnomatterhowselectedormatchedbecauseofimpossibilityincontrollingforallcharacteristicsthatmayaffectthecondition.Somematchingcriteriamightbeincorrect,forexample,orothersmaybeexcluded.Hereishowasurveycouldbeusedwithacasecontroldesign.

Example:CaseControlDesignTheUniversityMedicalClinicrandomlydivided10,000patientsbetween21and55yearsofageintotwogroups:thosewhohadhadamigraineinthepast12monthsandthosewhohadnot.Theyusedastratifiedrandomsamplingtechniquetoselect100peoplefromeachgroup,halfofwhomwerefemale.Asurveywasconductedtofindoutaboutthefollowinginrelationtoeachgroup:

TypicaldailyactivitiesPotentialsourcesofstressFamilyhistoryandbackgroundMedicalhistoryDiet

SurveyDesignValidityValidsurveyinformationcomesfromreliableandvalidsurveyinstruments(measurementvalidity)andfromthecontextinwhichthesurveytakesplace(designvalidity).Thetwotypesofvaliditydifferinthatmeasurementvalidityreferstothecharacteristicsofthesurveyinstrument,whereasdesignvalidityreferstotheselectionofsurveyrespondentsandthetimingandfrequencyofadministration..Measurementvalidityasksthefollowing:Isthesurvey’scontentcomprehensive?Doesgoodperformancepredictsuccesslateron?Doesameaningfuldifferenceexistbetweenhighandlowscorers?Designvalidityasksthis:Howoftenwillthesurveybegiven?When?Howwillrespondentsbeselected?

Becausesurveyorsworkinanimperfectworld,theircarefullydesignedstudiesareoftenthreatenedbyfactorsoverwhichtheyhavenocontrol.Somepeoplesaytheywilltakethesurvey,buttheydon’tortheydropoutmidway.Thesurveyoraimsforatrueexperimentbutdoesn’tquitemanageit.Thelistgoeson,andthethreatshavetheeffectofbiasingorprejudicingtheresults.Thesethreatsorbiaseslimitthe

survey’saccuracy.

CampbellandStanley(1963)identifiedandsummarizedthreatstoresearchdesigns.Theydividedthethreatsintothosethatunfavorablyaffecttheinternalvalidityofthedesignandthosethatunfavorablyaffecttheexternalvalidity.Adesignhasinternalvalidityifastudy’soutcome(say,animprovementinattitudetowardschool)isduetoorcausedbythevariablesthatarecontrolledormanipulatedinthestudy(suchasanexperimentalprogram).Itisexternallyvalidifthefindingsapplytootherpeopleandothersettings.

InternalValidityHereisareviewofthethreatstointernalvaliditythatCampbellandStanleyassembled.

ExternalValidityThreatstoexternalvalidityaremostoftentheconsequenceofthewayinwhichrespondentsareselectedandassignedtostudygroups.Forexample,respondentsinanexperimentalsituationmayanswerquestionsatypicallybecausetheyknowtheyareinaspecialexperiment.ThisiscalledtheHawthorneeffect.Externalvalidityisalsothreatenedwheneverrespondentsaretested,surveyed,orobserved.Theymaybecomealerttothekindsofbehaviorsthatareexpectedorfavored.HerearesomeofthethreatstoexternalvaliditythatCampbellandStanleyidentified.

Surveys,ResearchDesign,andInternalandExternalValidityWhenyouchooseandimplementasurveydesign,youalmostalwaysautomaticallyinheritthreatstointernalvalidity.Theoneexceptionistherandomizedcontrolledtrialsinwhichrespondentsarerandomlyselectedandassignedwithoutknowingwhethertheyareinanexperimentaloracontrolgroup(thedouble-blindrandomized-controlledtrial).Perfectrandomizedtrialsrequirevastresources.Althoughtheyareconsideredthegoldstandardformanyresearchstudies,theyaredifficultandcostlytoimplement,soyouprobablywillnotfindmany.

Lookatthisexampleforacross-sectionalsurvey,oneofthemosttypicalstand-alonesurveys.

Example:Cross-SectionalSurveyandThreatstoValidityAsurveyorwasaskedtodescribetheamountoftimephysiciansspendwiththeirolderpatients,notcountingofficevisits.Thesurveyorsurveyed16physiciansinanacademicgeriatricmedicalsettingduringthreerandomlysampled1-weekperiods.Usingastructuredform(aself-administeredquestionnaire),theresearchersfoundthatphysiciansspendaconsiderableamountoftimeprovidingcareoutsideofofficevisitsforpatientswithchronicillnesses.Theinvestigatorspointoutthatthestudy’svaliditymayhavebeenlimitedbythefactthatdatawereself-reported(measurementvalidity)andwereobtainedfromanacademicgeriatricmedicine(internalvalidity:selection).

Ifyouwouldliketolearnmoreaboutthissurvey,seeFarber,J.,Siu,A.,&Bloom,P.(2007).Howmuchtimedophysiciansspendprovidingcareoutsideofofficevisits?AnnalsofInternalMedicine,147(10),693–698.

Example:LongitudinalDesignandThreatstoValidityTheGreatSmokyMountainsStudyincludesarepresentativesampleofchildrenandadolescentsages9to16yearsfromthesoutheasternUnitedStates.Childrenandtheirparentsinthestudywereinterviewedonmultipleoccasionsfrom1993to2000aboutthesubjects’recentpsychiatricandsuicidalhistory.Thepurposesoftheinterviewsweretocharacterizerisksforsuicidalityassociatedwithanxiety,depressive,disruptivebehavior,andsubstanceusedisordersandtotestwhethertherearecriticalcombinationsofdisordersthatdiscriminateat-riskyouthindependentofseverityofpsychopathology.

Interviewerswereresidentsofthestudyareaandhadatleastabachelor’s-leveldegree.Theyreceived1monthoftrainingandconstantmonitoringforqualitycontrol.Interviewsusuallytookplaceathome.Theinvestigatorsfoundthattheseverityofsymptom-relatedimpairmentandtotalsymptomloadexplainedmostoftheriskforsuicide.Onlydepressionplusanxietydisorderdiscriminatedat-riskyouthindependentofseverityofpsychopathology.

TheinvestigatorsnotethatthesamplewasderivedfromarelativelysmallgeographicareainthesoutheasternUnitedStatesand

maynotberepresentativeofotherareas(internalvalidity:selection).Also,theydidnotmedicallyverifyreportedsuicideattempts(measurementvalidity:concurrentvalidity).

Ifyouwouldliketolearnmoreaboutthisstudy,seeFoley,D.L.,Goldston,D.B.,Costello,E.J.,&Angold,A.(2006).Proximalpsychiatricriskfactorsforsuicidalityinyouth:TheGreatSmokyMountainsStudy.ArchivesofGeneralPsychiatry,63(9),1017–1024.

Example:RandomizedControlledTrialsandValidityPeoplewhoareoncevictimsofinterpersonalviolenceoftenbecomevictimsagain,withthefrequencyofrecurrencerangingfrom6%to44%.Thisstudyhypothesizedthatthishighrecurrencerateamongyouthcanbereducediftheirunmetpsychosocialneedsaremet.Consentingyouthandtheirguardianswererandomlyassignedtoanexperimentalorcontrolgroup.Anumberofsurveyquestionnairesandscalesweredeveloped,includingParentingPracticesQuestionnaire,FamilyRelationsScale,andTheChildSelf-ReportofDelinquency.

Theinvestigatorsnotemanythreatstothevalidityofthestudythatlimitconfidenceinthefindings.Forexample,thestudydesignwasnotblinded,soparticipantsmayhavebehaveduncharacteristically(Hawthorneeffect).Almostallofthecomponentsofthelargersurveywerevalidated,butthecombinationwasnot(measurementvalidity).

Thestudywaslimitedbythoseyouthwhowereexcluded,losttofollow-up(internalvalidity:selectionandattrition),orhadincompletedocuments(measurementvalidity).

Ifyouwouldliketolearnmoreaboutthisstudyandthesurveysthatwereused,seeZun,L.,Downey,L.,&Rosen,J.(2006).TheeffectivenessofanED-basedviolencepreventionprogram.AmericanJournalofEmergencyMedicine,24(1),8–13.

SurveysWithQualitativeData:ThreatstoInternalandExternalValidityQualitativesurveyorsoftendifferfromtheirmorequantitativelyorientedcolleaguesinestablishingclose,sometimespersonal,relationshipswithrespondents.Qualitativesurveyorsmayspendmonthsorevenyearslivingincommunitiestofindoutfirst-handaboutinhabitants’needs,practices,andvalues.Theclosenessbetweensurveyorandrespondentgivesthesurveyorthepotentialforcollectingin-depthinformationonculturallyimportantandoftensensitivetopics.Atthesame,astrongconnectionmayintroducebiasintothesurveybecausetherespondentsknowtheyarepartofastudy(Hawthorneeffect),theymaybehavedifferentlyasaresultofansweringquestions(reactiveeffectsoftesting),andtheymayrespondtothesurveyor(appearance,perceivedattitude)ratherthanthesurvey.Atthesametime,thesurveyors’attachmenttothecommunitycanalsoaffectwhichquestionstheyask,theirchoiceofpeopletointerview,andtheirunderstandingandexplanationofthefindings.

Herearefivepotentialthreatstothevalidityofqualitativesurveys:

1. Thedynamicsofthesurvey-respondentrelationshipcaninfluencewhoisselectedandwhorespondstothesurvey,anditcanaffectthetruthfulnessoftheresponses.Thisissimilartoselectionbias,whichaffectsbothinternalandexternalvalidity.

2. Thesurveyors’preferenceforcertaintheories,designs,orissuescaninfluencethechoiceofsurveytopicsandtheirexplanationofthefindings.

3. Therecanbebiasduetohowandwhichevents,places,andpeoplearesampled.4. Financialandhumanresourcescanaffecttheavailabilityandreliabilityofinformants.5. Thesurveyors’personalandprofessionalqualities,includingtheirexperienceandrelationshiptothe

communityandindividualpeople,canaffectallcomponentsofthesurveyincludinghowitis

formatted;thesamplingmethodsusedandthequestionsasked;whocollects,analyzes,andreportsthefindings;whenthefindingsarereleased,towhom,andinwhatform(written?oral?online?).

Example:QualitativeStudiesandValidityForoveradecade,improvingAfghanwomen’shealthandreducinghighmaternalmortalityhasbeenapriorityoftheAfghanMinistryofPublicHealthanditsinternationalpartners.Aqualitativeethnographicstudyinonematernityhospitalaimedtounderstandtheperspectivesofhealthcareprovidersontheirroles,experiences,values,andmotivationsandtheimpactofthesedeterminantsonthecareofperinatalwomenandtheirbabies.

Theresearchersspent6weeksofobservationfollowedby22semi-structuredinterviewsandfourinformalgroupdiscussionswithstaff,twofocusgroupdiscussionswithwomen,and41backgroundinterviewswithAfghanandnon-Afghanmedicalandculturalexperts.

Thestudyfindings,accordingtotheresearchers,werelimited[biased]bythefactthatthatpeoplewhowerewillingtotalktotheinvestigatorswerepossiblythemorevulnerablestaffandthiscanhaveskewedthefindings(selection).

Thestudywasconductedwiththeassistanceofatranslator.Theresearchersnotethatalthougheveryeffortwasmadetoreducethenegativeimpactoftranslationandensurequality,someofthedepth,richness,andculturalnuanceswerelost.Somequestionsweresimplifiedandsomesensitivequestionswerenotaskedtoavoidmisunderstanding(measurementvalidityiscompromised).

Afinallimitationwasthatholdinginterviewsinhospitalwasnotidealassomeinterviewsweredisturbedbythedemandsofthehospital.Itwasnotpossibletoconductinterviewsatadifferentlocationortime,duetothebusylivesandfamilyrestrictionsonthemovementsofmanyparticipants(measurementvalidityiscompromised).

Ifyouwouldliketolearnmoreaboutthisstudyandthesurveysthatwereused,seeArnold,R.,vanTeijlingen,E.,Ryan,K.,&Holloway,I.(2014).UnderstandingAfghanhealthcareproviders:AqualitativestudyofthecultureofcareinaKabulmaternityhospital.BJOG:AnInternationalJournalofObstetrics&Gynaecology.

SummingUpAsurvey’sdesignreferstothefrequencyofitsadministration(onetimeormoreoften),itssequence(justbeforeandimmediatelyafteranewprogram),andthenumberofgroupsinvolved(allstudentsinoneschooldistrictorallstudentsineachof10districts)andhowgroupsareassigned(atrandomorconveniently).Surveydesignsareobservationalorexperimental.Cross-sectionalsurveysareobservationalstudiesthatproducedatadescribingthingsastheyareandarerelativelyeasytoimplement.Ifthingschangerapidly,thesurveyinformationwillpossiblybecomeoutdated.Longitudinalsurveysprovidethesurveyorwithdataonwhethereffectslastovertime.Unlesstherespondentsaremotivated,theymaydropout,andtheremaindermaynotbetypicaloftheentiregroup.Also,thegroupmaybeatypicalinthattheywerewillingandhadthetimetoparticipateintheprogramandcompletethesurveys.Trueexperimentsarecharacterizedbyrandomselectionandassignment.Manyresearchersassertthatthebesttrueexperiments—theleastbiased—arethoseinwhichparticipantsandinvestigatorsdonotknowtheirgroupassignment.Implementingtrueexperimentsmaybedifficultinthe“real”worldofsurveysandsocialresearch.Theyarerelativelycostlyandrequiretimeforimplementationandanalysis.Nevertheless,ifyouwanttoknowwhetheranexperimenthasresults,thetrueexperimentorrandomizedcontrolledtrialisthewaytogo.Quasi-experimentaldesignsareeasiertoimplement.However,becausegroupsarenotrandomly

constituted,theirinitialdifferencesmayberesponsiblefortheoutcome.Normativeorhistoricalsurveydesignscanbelessexpensiveandtime-consumingthanothercomparisondesignsifyoucanfindappropriatedatabasesforcomparisonpurposes.Validsurveyinformationcomesfromreliableandvalidsurveyinstruments(measurementvalidity)andfromthevalidityofthesurvey’sdesign(designvalidity).Thebestsurveydesignsminimizesourcesofbiasbycontrollingselection,usingrandomassignmenttogroup,andblindingparticipantsandresearchers.Measurementanddesignvaliditydifferinthatthefirstreferstothecharacteristicsofthesurveyinstrument,whereasthesecondreferstothecontextinwhichthesurveytakesplace.Measurementvalidityasksthefollowing:Isitscontentcomprehensive?Doesgoodperformancepredictsuccesslateron?Doesameaningfuldifferenceexistbetweenhighandlowscorers?Designvalidityasksthis:Howoftenwillthesurveybegiven?When?Howwillrespondentsbeselected?Adesignhasinternalvalidityifastudy’soutcomeisduetoorcausedbythevariablesthatarecontrolledormanipulatedinthestudy.Itisexternallyvalidifthefindingsapplytootherpeopleandothersettings.Threatstointernalvalidityincludeselectionofparticipants,history,maturation,testing,instrumentation,andstatisticalregression.Threatstoexternalvalidityincludetheinteractioneffectsofselectionbiasesandtheexperimentaltreatment,thereactiveeffectsoftesting,thereactiveeffectsofexperimentalarrangementortheHawthorneeffect,andmultipleprograminterference.Qualitativesurveysmayexperiencethreatstotheirvalidityresultingfromthesurveyors’closenesstothecommunitiesandindividualsbeingsurveyedaswellasfromthesurveyors’choiceofevents,people,andquestions.

ThinkAboutThis1. Findthefollowingarticles(oranyfourcontainingsurveys),nametheresearchdesignusedforthe

survey,anddiscussthedesign’sinternalandexternalvalidity.ArticlesArnold,R.,vanTeijlingen,E.,Ryan,K.,&Holloway,I.(2014).UnderstandingAfghanhealthcareproviders:AqualitativestudyofthecultureofcareinaKabulmaternityhospital.BJOG:AnInternationalJournalofObstetrics&Gynaecology.Gardner,F.,Burton,J.,&Klimes,I.(2006).Randomisedcontrolledtrialofaparentinginterventioninthevoluntarysectorforreducingchildconductproblems:Outcomesandmechanismsofchange.JournalofChildPsychologyandPsychiatry,47(11),1123–1132.Gatto,N.M.,Ventura,E.E.,Cook,L.T.,Gyllenhammer,L.E.,&Davis,J.N.(2012).LASprouts:Agarden-basednutritioninterventionpilotprograminfluencesmotivationandpreferencesforfruitsandvegetablesinLatinoyouth.JournaloftheAcademyofNutritionandDietetics,112(6),913–920.Patrick,K.,Calfas,K.J.,Norman,G.J.,Zabinski,M.F.,Sallis,J.F.,Rupp,J.,etal.(2006).Randomizedcontrolledtrialofaprimarycareandhome-basedinterventionforphysicalactivityandnutritionbehaviors:PACE+foradolescents.ArchivesofPediatricAdolescentMedicine,160(2),128–136.

2. Explainhoweachoneofthesetermsisusedwhentalkingaboutsurveys:researchdesign,measurementvalidity,designvalidity,internalvalidity,externalvalidity,cross-sectionalsurvey,longitudinalsurvey,cohortdesign,casecontroldesign,trueexperiment,randomizedcontrolled

trial,andquasi-experiment.

ReferencesCampbell,D.,&Stanley,J.(1963).Experimentalandquasi-experimentaldesignsforresearch.Chicago,IL:Rand-McNally.

Gardner,F.,Burton,J.,&Klimes,I.(2006).Randomisedcontrolledtrialofaparentinginterventioninthevoluntarysectorforreducingchildconductproblems:Outcomesandmechanismsofchange.JournalofChildPsychologyandPsychiatry,47(11),1123–1132.

Patrick,K.,Calfas,K.J.,Norman,G.J.,Zabinski,M.F.,Sallis,J.F.,Rupp,J.,etal.(2006).Randomizedcontrolledtrialofaprimarycareandhome-basedinterventionforphysicalactivityandnutritionbehaviors:PACE+foradolescents.ArchivesofPediatricAdolescentMedicine,160(2),128–136.

6AnalyzingandOrganizingDataFromSurveys

OverviewAnalyzingdatameansusingstatisticalandqualitativemethodstodescribeandinterpretrespondents’answerstothesurvey’squestions.

Surveyorsalmostalwaysreportdescriptivestatistics,suchastherespondents’averageage,educationallevel,andgradeorscore.Sometimestheysummarizethedataasacross-tabulation(cross-tab)todescribetherelationshipbetweentheanswerstotwodifferentsurveyitems.Supposeasurveyorintendstodescribehow100boysand100girlsrespondtoasurveyoftheirsatisfactionwithanewschool-basedsportsprogram.Across-taboftheresultscouldtakethisform:

Thetableshowsthat55%ofgirlsaresatisfied,butonly42%ofboysare.Arethe13percentagepointsameaningfuldifference?Onewayoffindingoutistoapplyastatisticaltestsuchasachi-squaretest.Chi-squaretestsevaluatestatisticallysignificantdifferencesbetweentwoormoregroupswhenthedataareproportions.Ifthedataarecontinuous,thesurveyorusesotherstatisticalteststoevaluatesignificance.Onesuchisthettest.Choosingastatisticaltestalwaysnecessitatesunderstandingwhetherthedataarecategoricalorcontinuous.Categoricalandcontinuousdataresultfromthetypeofresponsescalethataccompanieseachquestion.

Statisticalmethodsallowsurveyorstomakestatementsaboutstatisticalsignificance,onemeasureofthemeaningfulnessofsurveyresults.Afindingofstatisticaldifferencesamongtwogroups’averagescoresimpliesthatthereisindeedadifferencebetweenthemandthat,mostimportant,thedifferencesare“real”becausetheyareunlikelytohaveoccurredbychance.Butsurveyorsarealsointerestedinfindingoutwhetherthefindingsaremeaningfultothepeoplewhousetheresults.Confidenceintervalsprovidearangeofvaluesandhelpindecidingwhetherdifferencesarepracticallymeaningful.

Inadditiontodescriptionsofresponsesanddifferencesinresponses,surveyorsarealsoconcernedwiththerelationshiporcorrelationamongtheanswerstoquestions.Ahighcorrelationbetweenheightandweight,forexample,suggeststhattallerpeopleweighmoreandthatheavierpeoplearetaller.Regressionsareastatisticaltechniquethatusescorrelationsasthebasisforpredictingthevalueofonevariablefromtheother.Forexample,supposeyouwanttoknowwhetherbeinginvolvedinhighschoolathleticsisassociatedwith(orpredicts)participationincollegeathletics.Tofindout,youwoulduseregressionanalysiswithyoursurveydata.

Othertechniquesthatsurveyorsuseincluderisksandodds.Thesestatisticaltechniquesproduceinformationonthelikelihoodthataparticularoutcomewilloccurwithinagroupandcanalsobeusedtocompareoutcomesamonggroups.

Contentanalysisisaqualitativemethodthatcanbeusedtofindcommonideasorthemesintheanswerstoopen-endedsurveyresponses.Contentanalysisrequirescodingcommentsandresponsestoopen-endedquestions.Ifalargeenoughnumberofpeopleconsistentlywriteinthattheyareconcernedwiththesameproblemorexpressaparticularpointofview,itbecomesatheme.

Datamanagementisanintegralcomponentofdataanalysis.Managingdatameansorganizingdataintovariablesandnamingandcodingthemsothatoncetheyareinadatabasetheycanbeanalyzed.Assigningcodestodataisanessentialanalyticactivity.In

largesurveys,youmustverifythereliabilityofthecoding.Twocodersareoftenused.Tofindoutwhethercodingisreliable,considercomputingastatisticcalledthekappa.Thekappaprovidesameasureofagreementbetweentwocoders.

Youmustreviewsurveyreturnsformissingdataanddecidehowtohandlethem.Missingdataresultfromunansweredquestionsorentiresurveys.

Dataentryistheprocessofgettingthesurvey’sresponsesintoadatabase.Onlinesurveydataisautomaticallyenteredintoadatabase,withtheluxuryofreceivingreportsinrealtime.Surveydataalsomaybescannedinorhand-enteredintospreadsheetsforsubsequentconversionintoadatabase.Regardlessofhowthedataareentered,theresponsesneedtobecheckedfirstforvalidity.Thesurveyordoesnotusuallywantanswersfrommenaboutthephysicaljoysofgivingbirth,forexample.Mostonlinesurveyprogramshavetheoptionforautomaticvaliditychecking.Acleandatasetisonethatcanbeusedbyanyonetogetresultsthatareidenticaltoyours.Databecome“dirty”foranumberofreasons,includingpoorquestions,misleadingresponses,miscoding,incorrectdataentry,andmissingresponses.Toavoiddirtydata,makesurethatcodersordataenterersareexperienced,welltrained,andsupervised.Ifyouareusingonlinesurveysoftware,youcanfindavalidationfunctionthatwillallowyoutoguidetheresponsessothattheyareuniformandmeetyourstandards.Thiswillhelpensureacleandataset.

WhatIsTypicalAnyway?SomeCommonlyUsedMethodsforAnalyzingSurveyDataThedayhasarrived.Allthesurveyshavebeenreturned.Theresponserateishigh.Allthequestionsandalltheformshavebeenfilledout.Nowisthetimetofindoutwhatthesurveyshows:Howmanymenresponded?Howmanywomen?Dotheydiffer?Havetheirviewschangedinthepast5years?Surveyorsanswerquestionslikethesebyanalyzingsurveyresponsestoobtainresponsefrequenciesandaverageresponsesandtolookforrelationships,differences,andchanges.Someanalysismethodscommonlyusedinsurveysincludethefollowing.

DescriptiveStatisticsThesearethebasicunitsfordescribingasurvey’srespondentsandresponses.Descriptivestatisticsprovidesimplesummariesaboutthesampleandtheresponsestosomeorallquestions.Togetherwithsimplegraphicsanalysis(suchaspiechartsandbargraphs),theyformthebasisofpracticallyeveryquantitativeanalysisofsurveydata.Descriptivestatisticsforsurveysincludefrequenciesorfrequencydistributions(numbersandpercentages),measuresofcentraltendency(themean,median,andmode),andmeasuresofvariation(rangeandstandarddeviation).

Frequencies

Afrequencyisacomputationofhowmanypeoplefitintoacategory(e.g.,20aremorethan55yearsofage;150haveacold)orchoosearesponse(e.g.,32saiddefinitelyveryimportant;8saidmorethanthreetimesaweek).

Considerthis:50preschooldirectorsaresurveyed.Allschoolsarepubliclysupported.Thesurveyfollows.

Example:PreschoolPurposesQuestionnaireBelowaresomepossibleobjectivesofpreschooleducation.Circlehowimportanteachisinguidingyourschool’sprogram.(Number1beingdefinitelyimportantandNumber5beingdefinitelynotimportant.)

Forthepreschoolpurposesquestion,youcansummarizetheresponsesasshowninthefollowingexample.

Example:FrequencyofQuestionnaireResponses

Sometimesyouwanttogrouptheresponsestogether(or,intechnicalterms,prepareafrequencydistributionofgroupedresponses),asshowninthenextexample.

Example:GroupedRatingsofPreschoolPurposesby50Directors

Inthistable,theresponsesaredividedintotwocategories(dichotomized):important(includingtwocategoriesofresponse:definitelyimportantandimportant)andnotimportant(includingtwocategoriesofresponses:definitelyunimportantandunimportant).Whygroupresponsestogether?Ifonlyafewrespondentsselectoneofthechoices(sayonlythreeanswerdefinitelyimportant),thenthecategorymayloseitsmeaningtoyou.

Averages:Means,Medians,andModesThemean,median,andmodeareallmeasuresofaverageortypicalperformance.

Themean

Thearithmeticaverage,themean,requiressummingunitsanddividingbythenumberofunitsyouhaveaddedtogether.Herearefivescores:50,30,24,10,and6.

Theaverageforthesefivescoresis50+30+24+10+6dividedby5=24.

Theformulaforthemeanismx=SxN.

μisthesymbolforthemean,and∑standsforthesum,so∑xmeanstoaddallthenumericalvaluessuchasx.Nstandsforthenumberofx’s.

Supposeyouwantedtocomputetheaverageratingforthefourpreschoolpurposes.First,youwouldhavetoknowhowmanydirectorsassignedeachratingtoeachpurpose.LookatTable6.1.

Tocomputetheaverageratingforeachpurpose,youwouldmultiplythenumberofdirectorswhochoseeachpointonthescaletimesthevalueofthescale.Forcreativity,13directorschosea1,soyouwouldmultiple13×1,addtheresultstogether(13×1)+(2×2)+(5×3)+(20×4)+(10×5),anddividebythenumberofdirectors:13+4+15+80+50dividedby50=3.24,theaverageratingforcreativity.

Inthiscase,theclosertheaveragesareto1,themoreimportantthepurpose.Lookatthistable.

Themedian

Themedianisthepointonascalethathasanequalnumberofscoresaboveandbelowit.Anotherwayofputtingitisthatthemedianisatthe50thpercentile.Becausethemedianalwaysfallsinthemiddle,itisusedwhenyouwanttodescribetypicalperformance.

Whydoyouneedtypicalperformance?Supposeyouhadasetofscoreslikethis:5,5,6,6,6,8,and104.Theaverageis20,andthemedianis6.Howcanthishappen?Itmayoccurifthegroupyouaresamplingwasdividedinitsattitude(orknowledge,health,etc.),withmostpeoplefeelingonewayandsomefeelingmuchdifferent.Itcanalsohappenifyouareunabletocollectallthedatayouplanto,andmanyofthepeoplewithoneviewarenotrepresentedintheresponses.

Hereishowtocomputethemedianifyouhaveanequalnumberofscores:

1. Arrangethescoresinorderofsize.2. PlacethemedianbetweentheN/2scoreandthe(N/2)+1score(whereNequalsthenumberof

scores),usingthearithmeticaverage.

Example:ComputingtheMedianforanEvenNumberofScoresTakethesescores:−2,0,6,7,9,and9.

TherearesixscoressoN=6,withN/2=3and(N/2)+1=4.Thethirdscoreinorderequals6andthefourthequals7,sothemedianis6.5.

Takethesescores:2,4,5,8,9,and11.

Again,N=6,sothemedianisbetweenthethirdandfourthscores,5and8.Thistime,however,thereisagapofthreeunitsbetweenthethirdandfourthscores.Addingthetwoscores5+8anddividingby2givesavalueof6.5forthemedian.

Whatasetofdataissmallandoddinnumber,dothis:

1. Arrangethescoresinorderofsize.2. Placethemedianatthescorethatisthe(N+1)/2fromthebottomofthedistribution.

Example:ComputingtheMedianforanOddNumberofScoresTrythesenumbers:−9,−8,−6,−6−4,−3,−2,0,and2.

Themedianoftheseninescoresis(9+1)/2orthefifthscore,andsothemedianis−4.

Butsupposethatineitheraneven-numberedorodd-numberedsetofcases,youhaveseveralidenticalscoresatthemedianorborderingit,aswiththisset:3,6,7,8,8,8,9,9,10,and12.

WhenN=10,asinthiscase,themedianwouldusuallyfallbetweenthefifthandsixthscore.Calling8themedianthistimewillnotworkbecauseyouneedfivedifferentscoresaboveandfivedifferentscoresbelow.

Butifyouconsiderascoreof8aspartofacontinuumthatisevenlyspreadthroughouttheintervalbetween7.5and8.5,youcaninterpolateandcomeupwithanintermediatescore.

Thinkofitthisway:Youhavethreecases(3,6,and7)untilyoucometotheintervalof7.5to8.5,whichcontainsthree8s.Withinthatinterval,youwillneedtwomorecasestomakeatotalof5.So,youaddtwothirdsofthedistance,or6.7to7.5,andyougetamedianof8.17.

Themode

Themodeisascore(orapointonthescorescale)thathasahigherfrequencythanotherscoresinitsvicinity.Lookatthesescores.

DistributionAhasasinglemodeof29,with18peoplegettingthatscore.Thisdistributioniscalledunimodal.DistributionBhastwomodes,at25and31,sothedistributioniscalledbimodal.(Althoughthefrequencyof11associatedwiththescoreof31isthesameorlowerthanthatofotherscores,thescoreisamodebecauseithasahigherfrequencythantheothersnearit.)

Themode,withitsconcentrationofscores,describestheprevailingview.Youmightusethemodewhenyoususpectthatyouhaveagroupwithwidelydifferingcharacteristics.Forexample,ifyoususpectthatthereasonpeopleinProgramDdidbetterthanthoseinProgramCisbecausetheywereeconomicallybetterofftobeginwith,youmightcomparetheirincomesbeforetheyenteredeachprogram.

Ifyoufindsomethinglikethis,youcouldconcludethatyouareprobablyright.

PeopleinProgramDareonthehigh-incomepartofthescale,whereasthoseinProgramCareonthelow-incomeside.Together,GroupsCandDgiveyouabimodaldistribution.

Variation:Range,Variance,andStandardDeviationWhenyoucomputeanarithmeticaverage,everynumbercounts.Ifoneortwopeoplehaveveryhighorlowscores,theycanmaketheaverageseemartificiallyhigherorlower.Usingthemedianhelps,butittoocanbemisleading.Considerthissetofscores:2,3,4,5,6,7,and8.

Themeanandmedianare5.Ifyouweretochangethelasttwoscoresto17and18,themedianwouldstaythesameat5,withthreecasesaboveandbelowthescore.Themeanwouldriseto55/7=7.86.

Ifeverynumbercounts,itissometimesimportanttostudythespreadofscores—theirvariation—toshedlightonhowthemeancametobewhatitis.Areallthescoresnearthearithmeticaverage?Aresomeveryhighorverylow?Lookatthistable.

Ifyoulookattheaveragenumberofclassesattendedatthefourschools,youfindthatitis47.5.Lookingfurther,youseethat,atsomeschools,asfewas15classeswereattended,whereasatothers,all60were.Infact,itishardnottobestruckbytherangeofattendance.Atleasttwoinferencescanbemadebecauseyouhaveaccesstotherange:

1. Variationexistedinthreeofthefourschools(1,3,and4),withsometeachersattendingmanyclassesandothersattendingveryfew.

2. Inonlyoneschool(2)didalmosteveryoneattendaboutthesamenumberofclassesaseveryoneelse.

Insomecases,variationisconsideredanasset.Aprogramtotrainpeopletothinkindependentlyorcreativelymightexpectasurveytorevealavarietyofperspectives.Youalsoneedvariationtomakecomparisons.Ifeveryoneperformsequallywellorsharesthesameviews,youcannotselectthebest,thestrongest,themostliberalorconservative,andsoon.Inothercases,however,variationisadisappointmentbecauseyouwanteveryonetosharethesamevieworachieveaskillatthesamelevelofproficiency.Ifthedistrictthatissponsoringthecontinuingeducationbelievesitsprogramsareworthwhile,thewiderangeofattendancewillbedisappointing.

Anothermeasureofvariationiscalledthevariance,anditssquarerootiscalledthestandarddeviation.Thelatterisastatisticaltermbasedonascore’sdistancefromthemean.Infact,thestandarddeviationistheaveragedistancetheaveragescoreisfromthemean.Sometimes,insteadofthestandarddeviation,thevarianceisused.Thevarianceissimplythesquareofthestandarddeviation.

Youcanonlycomputethevarianceandstandarddeviationfromcontinuousdatasuchasaveragescores.Usetherangewhenyouwanttostudyvariationincategoricaldata(frombothcategoricalandordinalscales).

CorrelationandRegressionDatathatarecorrelatedarerelated.Ahighcorrelationbetweenheightandweight,forexample,suggeststhattallerpeopleweighmoreandthatheavierpeoplearetaller.Correlationsarereportedwithinarangeof+1(perfectpositivecorrelation)to−1(perfectnegativecorrelation).

Whenhighvaluesononevariableoccursimultaneouslywithhighvaluesonanother,thetwovariablesaresaidtobepositivelycorrelated,andwhenhighvaluesononevariableoccurwithlowvaluesonanother,thetwovariablesaresaidtobenegativelycorrelated.Thecorrelationcoefficientissymbolizedasrandisusuallyreportedintwodecimalplaces.

ASpearmanrankordercorrelationisusedwithcategoricaldata—thatis,datathatcomefromcategorical

orordinalscales.Arankordercorrelationprovidesameasureforthedegreeofassociationorequivalencebetweentwosetsofranks,asinthisexample.

Example:RankOrderCorrelationAclassof50collegestudentstakestwoattitudesurveys.Thefirstsurveypollstheirviewsonuniversalhealthcare,andthesecondsurveyasksabouttheirpoliticalpreferences.Johnscoreshighest(bestattitude)amongtherespondentsononemeasureandaverageonthesecond,Jane’sscoresarethe5thand8thhighest,Bill’sscoresarethe14thand13thhighest,andsoon.Arankordercorrelationcoefficientiscomputedtoseewhetherthetwosurveysagree:Dopeoplewhorankhighononesurveyalsorankhighontheother?

Whichstatisticalprocedureisanappropriateonetouseifthesurveyproducescontinuousdatasuchasheightandweight?ThePearsonproduct-momentcorrelationisusedtoestablishrelationshipsbetweentwosetsofcontinuousdata.Herearetwosituationsinwhichthesecorrelationsareappropriatelyused:

Therelationshipbetweengradepointaverages(GPAs)andscoresonanattitudetowardschoolsurveyiscorrelated.Therelationshipbetweenliberalandconservativeviews(1=liberalto10=conservative)andfamilyincome(from$5,000to$50,000)iscorrelated.

Regressionsusecorrelationsasthebasisforpredictingthevalueofonevariablefromtheother.Forexample,supposeyouwanttoknowwhetherstudentswhoreportbeinginvolvedinhighschoolathleticsareassociatedwith(orpredict)participationincollegeathletics.Tofindout,youcoulduseregressionanalysiswithyoursurveydata.

Asanotherexample,supposeyouwanttofindoutwhetherstudentswhohavehighgradesinhighschoolarealsolikelytohavehighgradesincollege.Inotherwords,youwanttoknowwhetherhighgradesinhighschoolpredicthighgradesincollege.Tofindout,youwouldfirstsurveycollegefreshmentogettheirhighschoolandfreshmanGPAs.Thenyouwouldexaminethecorrelationbetweenthetwoaverages.Finally,youwouldsurveyanewsetofhighschoolstudents(whoarecollegeboundbutnotthereyet)and,usingregressiontechniques,predicttheirfreshmanGPAbasedontheoriginalcorrelation.

Warning:Youcanusecorrelationstoidentifyrelationshipsbetweenvariables,butyoucannotusethemtoestablishcausation.Supposeyouinterviewasampleofpeopletofindouttheirhighestlevelofcompletededucation(Variable1).Youalsoaskthemtotellyouhowmanymagazinestheysubscribetothisyear(Variable2).Ananalysisofthecorrelationbetweeneducationandsubscribingpracticemayrevealthatthepeopleinyoursamplewhocompletedmanyyearsofschoolingalsosubscribetomanymagazines,butitcannotprovethatcompletingmanyyearsofschoolcausedthepracticeofsubscribing.Itmaybethatpeoplewhocompletethemostschoolingarealsothewealthiestinthesampleandthatenablesthemtosubscribe(theycanaffordit),oryoursamplemaybeatypicalortoosmalltodrawconclusions.

Multipleregressiontechniquesletyouconsidermorethanonepredictor.SupposeyoureviewtheeducationliteratureandfindthatstudentswhohavehighGPAsinhighschoolcomefromhomeswithrelativelyhighhouseholdincomes.Thatis,theliteraturesupportsthehypothesisthatahighcorrelationexistsbetweenhouseholdincomeandstudents’highschoolGPA.Youcannowaskthis:Whatarethecontributionofgradesinhighschool(Predictor1)andthecontributionofhighhouseholdincome(Predictor2)tocollegeGPA?Amultiple-regressionanalysismaybeusedtohelpyouanswerthisquestion.

DifferencesBetweenGroups

Descriptivedifferences:Thecross-tabulation

Across-tabulation(oftenabbreviatedascross-tab)isamethodusedtodescribetwovariablesatthesametime.Itisoftendisplayedasatable.Thetable’snameisusuallydefinedbythenumberofitsrowsandcolumns.

Thefollowingisafictitiousexampleofa3(rows)×2(columns)cross-tabdescribingtheresultsofasurveytofindoutabouttheuseofvariouselectroniccommunicationmethods.Inthisexample,thesurveyorscreateatablethatdescribestheanswerstotwosurveyquestions:

1. Onatypicalday,howmuchtimedoyouspendontheInternet?Theresponsechoicesarelessthan1hour,2−3hours,andmorethan2hours.

2. Whatisyourfeelingaboutsurveys?Theresponsechoicesarelovesurveysandhatesurveys.

Eachcellinthetablegivesthepercentageofsubjectsthatsharethatcombinationoftraits.Forinstance,inthissurvey,70%ofpeoplewhosaytheylovesurveysspend1hourorlessontheInterneteachday.

Cross-tabsarefrequentlyusedbecausetheyareeasytounderstandandmayprovideinsightintothemeaningofasurvey’sresults.Fromthetable,itappearsthat60%ofpeoplewhoreportspendingmorethan3hourseachdayontheInternethatesurveys.Thesurveyorwhoisinterestedinconductinganonlinesurveymayfindthisaninterestingresult.

Nowconsiderthis.AmarketingresearchfirmisinterestedinsurveyingcustomersoftheMacWindowCompanytofindoutwhichproductsitscustomersarelikelytopurchaseinthenext12months.Thisinformationistobeusedtoplanthecompany’sinventory.Thatis,MacWindowwillstocktheproductsthatmostpeopleplantopurchase.MacWindowconductsanonlinesurveytofindoutaboutitscustomers’plans.AportionofthesurveyisreproducedhereinFigure6.1.

Thesurveyorsprepareacross-tabulationoftheresultstotwoquestions:“Doyouplantopurchaseanyoftheseinthenext12months?”and“Whatisyourage?”showninTable6.2.

Fromthesecross-tabbeddata,MacWindowcanseecertainpatterns.Forinstance,thelargestnumberofrespondentsisbetween24and29yearsofage,andaboutone-thirdofthemareplanningtobuyabikeinthenext12months.Thatisasubstantialproportionofpotentialcustomers,andsoMacWindowmaydecidetostockacompletevarietyofbicyclestomeetthosecustomers’needs.Thenextlargestrespondentgroupisbetween30and35yearsold.About23%ofthisgroupisplanningtobuyacamcorder,with32%indicatingtheywillpurchaseasnowboard.Infact,almostone-thirdofallrespondentsover30areplanningtobuyasnowboardinthenext12months.

Figure6.1MacWindow’sOnlineSurvey

TheMacWindowCompanycancreateadditionalcross-tabstoanswerotherquestionsthatmayconcernthecompany.Forinstance,thecompanymaydecidetolookattherelationshipbetweenageandincome.Doesincomerisewithage,oraretheiryoungercustomersfromwealthierhouseholdsthantheirolderones?Domalesandfemalesdifferintheirhouseholdincome?Inpurchaseplans?Isthereadifferenceinexpectedpurchasesbetweenhigh-andlow-incomehouseholds?Thesearegoodquestions,butunfortunately,across-tabcanonlydescribetheincomeandpurchaseplans.Itcannottellwhetheranyobserveddifferencesaremeaningfulinstatisticalterms.Tofindout,thesurveyormustrelyonstandardstatisticalmethods.

StatisticalDifferencesSupposeyouwanttoknowwhetheronegroupofsurveyrespondentsisdifferentfromanother.Aretheyhealthier?Morelikelytovoteinthenextelection?Moreemployable?Betterspellers?MorelikelytopurchaseMacWindowCompany’sbikesordigitalcameras?Methodsusedtotestfordifferencesbetweengroups’surveysincludestatisticaltechniquessuchasthechi-squaretest,ttest,andanalysisofvariance(ANOVA;therearemanyothers).

Statisticaltestsactuallyhypothesizethatthesurveyoutcomesarethesameacrossgroups.Forinstance,ifyouwanttocompareattitudescoresbetweenGroupsAandBandyouwanttouseastatisticaltestsuchasattest,youhypothesizethatthemeanscoresarethesame.Theideaistosetupastrictenvironmentthatputsthisassumptiontothetest.

Thetestofequality—therespondentgroupsarethesame—iscalledthenullhypothesis.Afterstatingthehypothesis,thestatisticalanalysisisperformed,astatisticisproduced,andthisservesasaguideinacceptingorrejectingthenullhypothesis.Ifthenullisrejectedinfavorofanalternativehypothesis(the

groupsaredifferent),thentheresultsareconsideredstatisticallysignificant.

Supposeyousurveypeopleonacruise,halfofwhomarepartofanexperimentalprogramcombiningtourismwitheducation.Youarefairlycertainthattheexperimentisworking,andtheexperimentalgroupismuchmoresatisfiedthanthecomparisongroup.Nevertheless,asascientificsurveyor,youstartoutwiththeassumptionthatbothgroupsareequallysatisfied.Then,youcomparethetwogroups’averagesatisfactionscoresusingastatisticalmethodcalledanindependentttest.Youfindthatyouarecorrect,soyourejectthehypothesisofnodifference.Thereisastatisticallysignificantdifferenceafterall.Becausetheaveragesatisfactionscoresoftheexperimentalgroupwerehigher,youconcludethattheexperimentisworking,youwererightinthefirstplace,andnowyouhavestatisticstobackyouup.Thekeytoyourconclusionistheideaofstatisticalsignificance.Theresultsarestatisticallysignificantiftheyareunlikelytohaveoccurredbychance.

StatisticalSignificanceSupposeyousurveytheattitudesoftwogroupsofstudents,oneofwhichisinanexperimentalreadingprogram.Alsosupposethattheexperimentalgroup’sscoresarepoorerthantheothergroup’s—say,by10points.Aretherelativelypoorerscoresduetochance,oristhenewreadingprogramresponsible?Aresultthatisunlikelytohappenbychancecanbecalledstatisticallysignificant.Howmuchofadifferencebetweenthetwogroupsisnecessarybeforeyoucaneliminatechanceasthemotivation?

Todeterminestatisticalsignificance,youmustrelyonsamplingtheory.Forexample,youaskaquestionsuchasthis:Whatistheprobabilitythatmytworandomsamplesofstudentsfromthesamepopulationwillproducemeanscoresthatdifferbyasmuchas,say,10points?Twentypoints?

Supposeyoudecidethatachancehappeningof1timein100isanacceptablerisk.Thispredefinedprobability(p<.01)iscalledthelevelofsignificance.Ifthedifferencesyouobserveoccurnomorethan1outof100times,youcanrejectthenullhypothesisofnodifferencebetweengroups.

Byconvention,surveyorsusuallyusethe.05,.01,.001,andsoonsignificancelevel,meaningthattheobserveddifferenceintheexperimentalandtraditionalprogramswillbeconsideredstatisticallysignificantifthedifferenceof10pointswouldoccurbychance(assumingthetwogroupsarerandomsamplesfromthesamepopulation)only5timesin100,1timein100,or1timein1,000.

Intestingstatisticalhypotheses,youmustestablishrulesthatdeterminewhenyouwillacceptorrejectanullhypothesis.Take,forexample,astatisticaltestofanexperimental(A)andcontrol(B)readingprogram,wherethenullhypothesisisthatthemeanreadingscoresforbothgroupsareequal.Whenyouapplyastatisticaltest(suchasthettest)tothedata,youdonotexpecttofindzerodifferencesinaveragescoresbetweenthetwogroups.Instead,therealquestioniswhetherthedifferencesaresosmallthattheycouldhaveoccurredsimplybychance.Whenyouselecttworandomsamplesfromthesamepopulation,youcanexpecttheirmeanscorestobeclose,butnotexactlythesame.

Itisuptoyoutodecidehowfarapartthescoresmustbebeforeyouaresatisfiedthatthedifferenceisnotjustanaccident.Youcouldchoose.10,.05,.01,or.001asthelevelofsignificancedependingontheamountoferroryouarewillingtotolerateinrejectingthenullhypothesis.Whythesefractions?Thesearetheonestypicallyusedbysurveyandresearchanalystsbytradition.

Ifyouselectthe.05levelofsignificance,thenabout5timesin100youwillrejectthenullhypothesiswhenitis,infact,correct.Thishappensbecauseyouarecomparingtworandomsamplesfromthesamepopulation,andtheprobabilitythattheywilldifferbychancealoneis5%(p<.05).ThissituationisknownasaTypeIerror.Itistheprobabilityofrejectingthenullhypothesiswhenitistrue.Thatis,youcommitaType1errorwhenyousaythatthereisadifferencewhen,infact,noneexists.

Ifthelevelofsignificanceis.01,thentheprobabilityofaTypeIerrorisonly1in100,or1%.YoucanselectalevelofsignificancethatwouldvirtuallyeliminatethechanceofaTypeIerror,butthereareseriousconsequences.ThelesslikelyyouaretomakeaTypeIerror,themorelikelyyouaretomakeaTypeIIerror.

ATypeIIerroriswhenyouacceptanullhypothesisthatis,infact,incorrect.Yousay,forinstance,thereisnodifferencebetweentwogroups,butactuallythereis.WhenyoumakeaTypeIIerror,thedifferencebetweenthetwogroups’meanscoresdoesnotfallwithintherejectionregion(say,p<.05).

Thepowerofastatisticaltestistheprobabilitythatthetestwillrejectafalsenullhypothesis(thatitwillnotmakeaTypeIIerror).Anotherwaytothinkaboutthisisastheabilityofatesttodetectaneffectgiventhattheeffectactuallyexists.Aspowerincreases,thechancesofaTypeIIerrordecrease.TheprobabilityofaTypeIIerrorisreferredtoasthefalse-negativerate(β).Therefore,powerisequalto1−β.

Poweranalysiscaneitherbebefore(apriori)orafter(posthoc)surveydataarecollected.Aprioripoweranalysisistypicallyusedtodetermineanappropriatesamplesizetoachieveadequatepower.Posthocpoweranalysisusestheobtainedsamplesizeandeffectsizetodeterminewhatthepowerwasinthestudy.

Aneffectsizehelpstodeterminewhetherastatisticallysignificantdifferenceisadifferenceofpracticalconcern.Itisagenerictermforanestimateoftheeffectofbeinginastudyandisoftendefinedasthedifferenceinmeansbetweenanexperimentandcontrolgroupdividedbythestandarddeviationofthecontrolorofbothgroups.Givenasufficientlylargesamplesize,itispossibletoshowthatthereisadifferencebetweentwomeansbeingcomparedouttosomedecimalposition.Theeffectsizehelpsinknowingwhetherthedifferenceobservedisadifferencethatmatters.

Surveyorswhoareinterestedinanalyzingdifferencesbetweengroupsoftenexpressthosedifferencesbyusingconfidenceintervals(CIs)ratherthanapvalue.ACIisanestimateofapopulationparameterlikeamean,butitisdescribedwithinanintervalorrange.

Supposeyousurvey300randomlyselectedstudentsatamajoruniversity.Youaskthemhowmuchtheyspentononlinepurchasesinthepastmonth.Theaverageamountforthesampleturnsouttobe$100permonth.Butitjustmaybethatthese300studentsaremuchsmallerInternetspendersthanaretheotherstudentsintheuniversity.Thepopulationmean(theaverageforalltheuniversity’sstudents)couldbecloserto$500,forexample.Asmallnumberofuniversitystudentsmaybespendingthousandseachmonth,andthatwillraisethepopulationmeanconsiderably.

Thesample’smeanissimplynotonethesurveyorcanbeabsolutelycertainof(unlessthesampleisthewholepopulation).Thelargerthesamplesize,themoreconfidentthesurveyorcanbethatthepopulationmeanliesfairlyclosetothesamplemean.

CIscometotherescuebecausetheyarebasedontheconceptofrepeatedsampling.IfsamplesofthesamesizearedrawnrepeatedlyfromapopulationandaCIiscalculatedfromeachsample,thensomefraction—say,95%—oftheseintervalsshouldcontainthepopulationmean.

LookatTable6.3.

Theresultsareintheformofatablewithdescriptivestatistics:themean,standarddeviation,standarderror,andtheCI.Group1hasameanscoreof11.TheCIforthatscoreis7.67−14.33.IfsamplesofthesamesizeweredrawnrepeatedlyfromthepopulationfromwhichGroup1wasdrawnandaCIwascalculatedforeachsample,then95%oftheseintervalsshouldcontainthepopulationmean,whichwouldbebetween7.67and14.33.

YoucanplottheCIsonagraph.Ifthemeansdonotoverlap,differencesexist.Ifthemeanofonegroupiscontainedintheintervalofthesecond,differencesdonotexist.Iftheintervalsoverlap,butnotthemeans,youcannottell.

LookatthisgraphfortheCIs.

Group2’smeanscoreiswithinGroup1’sCI.Group3’sintervaldoesnotoverlapwitheithergroup.Differencesinthemeanscanbeseen,andyoucanrejectthenull(thatthemeansarethesame).Asanaside,ifyouweretouseastatisticalmethodlikeANOVA,youwouldfindthatthepvalueandCIsagreeintheiroutcome.TheCIandparerelated.Iftheintervalcontains0,thenthepisnotsignificant.

CIsareregularlyusedtodescribedifferences.Theyarepreferredinmanyplacesbecausetheyenableyoutoseetherangeofvalues.

SurveyingDifferences:UsualMethodsHerearesomecommonlyusedtechniquesforevaluatingdifferencesamonggroupsor,moreprecisely,testingwhethertheyarethesame.

Chi-SquareThechi-squaretestisusedwithcategoricaldata.Itteststhehypothesisthatsurveydataexpressedasproportionsareequal.Aproportioniswhatyougetwhenyoufindouthowmanyofallpossiblepeopleansweracertainwayorhaveaspecificcharacteristic.Supposeyousample1,000peopleandfindthat

250haveblueeyes.Theproportionofpeoplewithblueeyesis250/1,000.Supposethat,ofthe250peoplewithblueeyes,60arewomen.Supposeyouwanttocomparetheproportionofmen(190/250)andwomen(60/250)withblueeyes.Youhaveanideathatproportionsaresignificantlydifferentfromoneanother.Inotherwords,theyaretrueandnotchancedifferences.Thestatisticaltestyouwouldusetocomparetheproportionisthechi-square.Thesymbolforchi-squareisχ2.Thefollowingisanexampleofhowachi-squareisused.

Example:Chi-SquareThePrisonOrganizationsurveys60imprisonedmenand40imprisonedwomentofindoutwhethertheyperceivetheprisonsystemasfairorunfair.Theresultsareputinatable.

Thetableshowsthat57menand32womenbelievethesystemisfair,and3menand8womenbelievethatitisunfair.Thechi-squarecanbeusedtotestthenull(nodifferences)hypothesisthattheproportionsofmenandwomenarethesameintheirviewsoffairness.Ifthenulliskept(retained),youconclude“nodifferences.”Thetableiscalleda2×2table.Itlooksjustlikeacross-tab,butinthiscase,astatisticaltestisbeingdoneonthedata.Thefirst“2”consistsofthecolumnsformenandwomen,andthesecond“2”istherowsforfairandunfair.

ThetTestThettestisalsousedtotestfordifferences.Itallowsyoutocomparethemeansoftwogroupstodeterminetheprobabilitythatanydifferencesbetweenthemarerealandnotduetochance.Statisticsbookswilltellyoutohaveatleast20to30respondentspergroupandcontinuousdatatouseattest.Youneedcontinuousdatatocalculatethearithmeticaverageormean.

Example:tTestHopeHospitalhasinitiatedagourmetmealplan.CharityHospitalsaysgourmetmealscosttoomuch.Dothetwohospitalsdifferintheirpatients’satisfactionwithmeals?Thesurveyorssurvey50patientsatHopeand60atCharity.TheyusetheSatisfactionScale,whichaskspeopletomarkwhereona10-pointcontinuousscaletheirsatisfactionlies.Thesurveyorscomputetheaveragesatisfactionateachhospitalanduseattesttocomparetheresults.

TheMann-WhitneyUTestTheMann-WhitneyUtest(alsocalledtheWilcoxonranksum)enablesyoutocomparetwoindependentgroupswhenyoucannotusethettest—say,becausethesamplesizeistoosmall.Thisstatisticalmethodisatestoftheequalityofthemedians.

Example:Mann-WhitneyUTestPartridgeElementarySchoolhad10fourth-gradestudentswithseverehearingimpairments.Fourofthestudentswereinanewprogramtoteachthemtospeakmoreclearly.Attheendofonesemester,studentsinthenewprogramwerecomparedwiththesixstudentsinthetraditionalprogram.Aspecialeducationexpertratedeachchild’sabilitytospeakonascalefrom1to20,with20representingclearspeech.Ifmorestudentshadbeenavailable,thesurveyormighthavechosenthettesttocomparethetwogroups,butafterconsideringthesmallsamplesize,hedecidedtousetheMann-WhitneyUtest.

ANOVAGroupmeansoraveragescanalsobetestedwithANOVA.Thismethodletsyoutestaveragesachievedbytwoormoregroups.Forinstance,youcantestfordifferencesinaveragescoresonastatewidetestforstudentsinExperimentalSchoolAversusExperimentalSchoolBversusControlSchoolC.

RisksandOddsRisksandoddsareusedtodescribethelikelihoodthataparticularoutcomewilloccurwithinagroup,ortheycanbeusedtocomparegroups.Supposethat,forevery100personswhohaveacold,20peoplealsohaveacough.Theriskofacoughwithacoldis20/100or.20.Theoddsofhavingacoughwithacoldarecalculatedbycomparingthenumberofpersonswith(20)andwithout(80)coughs,or20/80,whichis.25.

Whenyouuserisksandoddstocomparegroups,youcomparetherelativelikelihoodthatanoutcomewilltakeplace.Therelativeriskexpressestheriskofaparticularoutcomeintheexperimentalgrouprelativetotheriskoftheoutcomeinthecontrolgroup.Theoddsratioisadescriptionofthecomparisonoftheoddsoftheoutcomeintheexperimentalgroupwiththeoddsinthecontrolgroup.Youcanusetherelativeriskortheoddsratiowith2×2tables.Lookatthefollowingtableoutline.

Example:OddsRatioandRelativeRisk

Youcanaskthefollowing:Whataretheoddsofstillsmoking(orquitting)intheexperimentalgroupcomparedtothecontrol?Whatistheriskofstillsmoking(orquitting)intheexperimentalgrouprelativetotheriskofstillsmoking(orquitting)inthecontrol?

Therelativeriskandtheoddsratiowillbelessthan1whenanoutcomeoccurslessfrequentlyintheexperimentalthaninthecontrolgroup.Similarly,bothwillbegreaterthan1iftheoutcomeoccursmorefrequentlyintheexperimentalthaninthecontrolgroup.

Whyusetheoddsratioorrelativerisk?Whynotstaywiththechi-square?TheansweristheCI.CIscanbeplacedaroundoddsratiosandrelativerisks(butnotchi-squares),accomplishingthesameobjectiveasasignificancetest.Thatis,youcandetermine(bylookingattheintervals)iftheresultsarestatisticallysignificant.LookatTable6.4describingtheresultsofasurveyofInternetuseforhealthinformation.

Thetablehasfivecolumnsandfiverows.Thefirstcolumnconsistsofdemographics.Becausethetabledisplaysoddsratioresults,eachcharacteristicisassumedtobecomparedtoitsopposite:maletofemale,marriedtonotmarried,andsoon.EachcellcontainsanoddsratioandaCI.Inthefemalerowand“EverUsedtheInternetforHealthInfo”column,youwillfindthreebitsofinformation:1.23istheoddsratioand1.06−1.43istheCI.Lookingatthesenumbers,youcanseethatfemalesweresignificantlymorelikelythanmales(oddsratiogreaterthan1andno0intheCI)toreporteverusingtheInternetforhealthinformationwhencomparedwithnotusingtheInternet.Inaddition,moreeducationandhighercognitivescorescorrespondedtoincreasedoddsofonlineuse.

Surveyorshaveanarsenalofstatisticalmethodsthatmaybeappropriatefortheirsurveys.Supposeasurveyorisinterestedincomparingthesatisfactionofpeopleinanewcommunicationsprogramandthoseinatraditionalone.Thesurveyorhasanideathatpeoplewhohaveparticipatedinthetraditionalprogramforatleast3yearswillbelesssatisfiedthanparticipantswhohavejoinedmorerecently.Thatis,lengthofparticipation,calledacovariate,canaffectsatisfactioneventhoughitisnotrelatedtothenewprogram.ThesurveyorcandiminishtheeffectsofthecovariatebyusingastatisticalprocedurecalledANOVA.

Hundredsofsuchusefulstatisticaltechniquesareavailableforsurveyors.Specialformsofttestsand

ANOVAscanbeusedtomeasurechangeovertime.AdependentttestmeasureschangeinasinglegroupfromTime1toTime2.Arepeated-measuresANOVAcanbeusedtodetectchangesinoneormoregroupsattwoormoretimes.

WiththeMcNemartest,eachpersonactsashisorherowncontrol,andsmallsamplesandcategoricaldataareused.Forexample,youcanuseaMcNemartesttofindoutwhetheradifferenceexistsinthenumberofstudentswhodoanddonot(thecategory)choosecareersinsoftwareengineeringafter20haveparticipatedinacounselingprogram.

ToBeorNottoBe:StatisticianorQualitativeAnalyst?Surveyorsdonothavetobestatisticianstoconductsurveys.However,theymustunderstandenoughaboutstatisticsandstatisticalmethodstoaskastatisticianforhelpandtoproperlyusethefunctionsprovidedbyonlinesurveyvendors.Surveyorsshouldalsobeabletoevaluatetheadequacyoftheanalytictechniquesrecommendedandusedbyothers.

Notallsurveydataareamenabletoroutinestatisticalanalysis.Somesurveyinformationconsistsoftheanswerstoopen-endedquestionsandrespondents’comments.Theseanswersandcommentsmaybebriefortheymaybeminiessays.Thesedataareoftencalledqualitativeandarecontrastedwiththestatisticalorquantitativedatathatresultfromtheclosedquestionsassociatedwithmostsurveys.

Qualitativedataanalysiscanbecomplexbecausetheinterpretermustfiltertheresponsethroughhisorherownunderstandingoftherespondent’sintentions.Qualitativeanalysisisoftendonebytwoormorepeopletoeliminateanysingleinterpreter’sperspectiveandcomeupwiththetruth.Largesurveyswithmanycommentscanusequalitativeanalysissoftwaretohelp.

ContentAnalysis,Open-EndedResponses,andCommentsContentanalysisisamethodofanalyzingqualitativedataforthepurposeofdrawinginferencesaboutthemeaningofrecordedinformationsuchastheopen-endedresponsesandcommentsmadebysurveyrespondents.Forthesurveyor,itmeanscarefullyreadingthecommentstoidentifythepresenceofcertainwords,concepts,themes,phrases,characters,orsentencesinordertoquantifythem.Acontentanalysisoftheresultsofahypotheticalexampleofasurveytofindoutaboutthecomparativeeffectsofteasingboysandgirlsmighttakethefollowingform.

Example:HypotheticalContentAnalysis:TeasingBoysandGirlsAllparticipantswereaskedaseriesofquestions,including“Whatwereyouteasedabout?”“Howdidyoufeelaboutbeingteased?”“Howdidyourespondtobeingteased?”and“Didyouchangeanythingaboutyourappearanceasaresultofbeingteased?Ifyes,whatdidyouchange?”

Theresponsesweredividedbysex.Responseswithineachsexwerearrangedbyquestionnumber(e.g.,responsestothequestion,“Howdidyoufeelaboutbeingteased?”werearrangedtogetherforanalysis).Thethemesthatemergedfromresponsestoaspecificquestionwereexaminedtogetherandcomparedbetweenthetwogroups.Differencesinsubstancewerenotedtoassesspossibledifferencesbetweenthegroups.Asthecontentemergedfromthedata,participants’commentsonspecificissueswereincludedtosupportthefindings.

Girlsfrequentlyreportednegativeopinionsandattitudestowardbeingteased(41.3%),followedbynegativeemotions(28.3%),residualeffects(19.6%),socialaspectsofteasing(8.7%),andactiontakeninresponsetoteasing(2.2%).

Boysmorefrequentlythangirlsreportedsocialaspectsofteasing.Theirexperiencesweredescribedasplayfuliftheywereteasedbyfriends.Forexample,oneparticipantstated,“Itdepends;ifit’smyfriend,thenItookitasajoke,butifit’snotmyfriend,Ithinkit’smeanandhurtful.”Anothersaid,“Withmyfriends,IamfinewithbeingteasedbecauseIteasedthem.”

PuttingtheCartinFrontoftheHorse:SelectingAnalysisMethodsTheappropriateanalysismethodforsurveydatadependsonsamplesize,thesurvey’sresearchdesign,andthecharacteristicsandqualityofthedata.Ifsomeoneasks,aspeoplesometimesdo,“Whatisthebestmethodforanalyzingsurveydata?”replywithsevenquestionsthatmustbeansweredfirst.

SevenQuestionstoAnswerBeforeChoosinganAnalysisMethodQuestion1.Howmanypeopleareyousurveying?Samplesizeisanimportantconsiderationinselectingananalyticstrategy.Somestatisticalmethodsdependonrelativelylargersamplesthandosomeothermethods.Lookatthesesamples:

1. All500teachersintheschooldistrict2. Astratifiedsampleof50teachersintheschooldistrict.Thestrataordivisionsaremenandwomen,

aswellashighschool,juniorhighschool,andelementaryschoolteachers.3. SixteachersatHartSeniorHighSchoolandfiveatJamesSeniorHighSchool

Askthefollowing:Doyouhavesufficientdatatocomputemorethantallies,averages,andthevariation?YouwouldforExampleA,butyouprobablycouldnotforExampleC.

WhataboutExampleB?Supposeyouweresurveyingapopulationofteachersofwhomabout60%werewomenand40%weremen.Ifyoutookasampleof50andyourstratificationmethodwasasuccess,youwouldhave30(60%women×50)womeninonegroupand20(40%×50)menintheother.Accordingtothestatisticaltheoryonwhichthettestisbased,groupsof30and20arejustaboutlargeenough.

Considerthefollowingtable.

Ifyouwanttocomparemenandwomenteachersinelementaryschool,youwillhaveonly2menand15womenforyouranalysis(althoughforthetotalyouwillhave20menand30women).Thisleadstothesecondofthesevenquestionstoaskwhenselectinganalysismethods.

Question2.Areyoulookingforrelationshipsorassociations?

Example:RelationshipsandAssocia-tions

1. Whatistherelationshipbetweenvoters’politicalviews(asexpressedonthePoliticalViewSurvey)andnumberofyearsofformalschooling?ThePoliticalViewSurveygivescontinuousscores

rangingfrom10(liberal)to100(conservative).2. ArehighscoresonthePoliticalViewSurveyassociatedwithhighscoresontheAuthoritarian-

LibertarianInventory?Howaboutmediumandlowscores?

Inthefirstexample,thedegreeofassociationbetweenthetwovariables,politicalviewscoresandnumberofyearsofformalschooling,canbecomputedbyusingthePearsonproduct-momentcorrelationbecausecontinuousdataareavailable.Forthesecondexample,withranksofhigh,medium,andlow,theSpearmanrankordercorrelationisappropriate.

Question3.Willyoubecomparinggroups?

Example:ComparingGroups

1. FifteenteachersatHartSeniorHighwillbecomparedwith20atJamesSeniorHigh.2. Atotalof100maleandfemalesteacherfromthedistrict’shighschools,juniorhighschools,and

elementaryschoolswillbesurveyedinDecemberandJuneandcomparedatbothtimes.

Ifyouarecomparinggroups,statisticaltestssuchasthechi-square,ttest,ANOVA,oddsratioorrelativerisks,andMcNemartestscanhelpyoudecidewhetheranyobserveddifferencesareduetosomerealoccurrenceorchance.

Question4.Willyoursurveybeconductedonceorseveraltimes?Thisquestionistofindoutwhetherthesurveydesigniscross-sectionalorlongitudinal.

Example:LookingforChange

1. All500teacherswillbesurveyedinDecember.2. Asampleof100teacherswillbesurveyedinDecemberandagaininJune.

Figure6.2isawaytodisplaytheresultsofsurveysconductedseveraltimes.

Thegraphshowsthatin2002,GroupBhadlowerscoresthanGroupA.By2004,GroupBhadpassedGroupA.Bothgroupsincreasedtheirscoresovertime.Arethedifferencesbetweengroupssignificant?Arethedifferencesovertimesignificant?Thegraphcannottellus.Forstatisticalsignificance,youneedstatisticalmethods.

Figure6.2SurveyResults

Hereisanotherwaytodisplaychangesovertimeusingthesamedata:

Thislinegraphenablesyoutovisuallyinspecttheslopeatwhichthescoreschanged.Butlikethebargraph,youcannottelljustfromlookingatthegraphwhether,andtowhatextent,thechangesare

significant.

Question5.Arethedatarecordedasnumbersandpercentagesorscoresandaverages?Lookatthis:

1. Onesetofsurveydataconsistsofthenumberandpercentageofteacherswhoagreeordisagreewithstatementsaboutthedistrict’seducationalpolicies.

2. Onesetofdataconsistsofscoresonameasureofattitudes(liberalvs.conservative)tovariousapproachestowardeducation.

Figure6.3ChangesinAttitudesOverTime

Surveydatacanbeinnumbersandpercentages(categoricaldata)orscoresthatareamenabletothecomputationofaverages(continuousdata).Theanalytictechniquesyouusewitheacharedifferent.LookatTable6.5andTable6.6.

Comparingnumbersofpeoplewhowereacceptedintolawschoolfromdifferentprograms(categoricaldata)andcomparingthemeanscoresonanattitudeinventory(continuousdata)requiredifferentanalysistechniques.Thefirstcomparisonmightusethechi-square,forexample,whereasthesecondmightuseattest.

Question6.Howmanyindependentanddependentvariablesinterestyou?

Avariableisacharacteristicofinteresttoyoursurvey.Healthstatus,attitudetowardschool,self-efficacy,group(experimental,control),gender,jobsatisfaction,andqualityoflifeareallvariables.Independentvariablesareusedtopredictorexplainfindings.Theyusuallyincludedemographics(suchasage,ethnicity,gender,income,andwhethertherespondentisinanexperimentaloracontrolgroup).Dependentvariablesarewhatyouarelookingforinyoursurvey:knowledge,attitudes,andbehavior.

Supposeyouwanttoknowwhethermenandwomendifferintheirqualityoflife.Theindependentvariableisgender(menandwomen),andthedependentvariableisqualityoflife.Ifyouwanttocomparemenandwomenintheirqualityoflifeandhealthstatus,youwillhaveoneindependentvariable(gender)andtwodependentvariables(qualityoflifeandhealthstatus).Youaddanindependentvariablewhenyouaskthis:Howdomenandwomenintheexperimentalandcontrolteamscompareintheirqualityoflifeandhealthstatus?Theaddedindependentvariableis“team”(experimentalandcontrol).

Youneedtoknowthenumberofindependentanddependentvariablestoselectanappropriateanalysismethod.Considerthefollowingdialogue:

Surveyor:Iwanttocomparetwogroupsofcollegeseniorstofindouttheirlifestylepreferences.Therearemorethan300ineachgroup.

Analyst:Whatdoyoumeanbylifestylepreferences?Areyouplanningtocomparenumbersofseniorswhoprefercertainstylesoflife—say,ruralversusurban?Orareyoucomparingtheirscoresonasurvey?

Surveyor:SeniorsinbothgroupshavecompletedLIFEQUEST,a100-itemsurvey.Itproducesscoresfrom1to100.Higherscoresrepresentmorespirituallifestyles,whereasloweronesmeanmorematerialones.

Analyst:Ifyouhavescoresonacontinuum,youhavecontinuousdata.Withtwogroupsandcontinuousdata,attestsoundsright.

Surveyor:WhatmethodwouldbecorrectifIdecidedtocomparenumbers?

Analyst:Probablyachi-square.

Surveyor:SupposeIhadmorethantwogroupsandcontinuousdata?

Analyst:WemightconsiderANOVA.

Question7.Arethedataofhighquality?

Dataareofhighqualityifthesurveyisreliableandvalid,administeredtotherightnumberoftherightpeople,andrecordedaccurately.Ifthedataareincompleteanduntruthful,itprobablydoesnotmatterwhichanalysismethodyouusebecausetheresultswillnotmeanmuch.

DataManagementWhatisthebestwaytomakesurethedataarecomplete,reliable,andreadyforanalysis,interpretation,andreporting?Answeringthisquestionmeanslearningaboutdataorganizationormanagement.

Organizingormanagingsurveydataisanessentialpartofdataanalysis.Itisthepartoftheanalyticprocessinwhichdataareenteredintoadatabasesothatyoucangetresults.Butbeforedatacanbeentered,youhavetodecidehowtopreparethemsothatyougettheinformationyouneedinauseableform.

CreatingaCodeBookorOperationsManualThecodesaretheunitsthat“speak”tothestatisticalsoftware.Suppose5,200collegestudentscompleteanalcoholusesurveycalledtheCollegeAlcohol-RelatedProblemsSurvey(CARPS).Onequestionofconcerntoyouishowmanymalesreportdrinkingfivedrinksatonesittingeveryday.Tofindthisout,youhavetotellthecomputerwhichvariablestolookfor(e.g.,genderandquantityandfrequencyofalcoholuse).Youalsohavetotellthecomputerhowtodistinguishmalesfromfemalesandhowtocomeupwithanumberthatrepresentshowmucheachpersondrinksandhowoften.

Computerprogramsreadaboutvariablessuchasgenderandquantity/frequencythroughcodes.LookatthisexcerptfromtheCARPS.

Example:ExcerptFromtheCARPS,aSurveytoDetectBingeDrinkinginCollegeStudents

1. Areyoumaleorfemale?[SEX]Male □0Female □1

2. Howofteninthepast12monthshaveyouhadfourormoredrinksofalcoholatonesitting?[QFDRINK]

Chooseoneanswer

Dailyoralmostdaily □1Fourorfivetimesaweek □2Twoorthreetimesaweek □3Twotofourtimesamonth □4Onetimeamonthorless □5Never □0

Thecodesarethenumberstotherightandjustbelowtheresponseboxes.Youusestatisticalprogramstotellyouhowmanypeoplewhoanswered0toQuestion1alsoanswered1toQuestion2.Todothis,thecomputermustbetoldthenameofthevariables(Question1=SEXandQuestion2=QFDRINK)andtheirvalues(0=maleand1=femaleand1=dailyoralmostdailyto0=never).

The“words”inbrackets(SEXandQFDRINK)correspondtothevariablesrepresentedbyeachquestion.Allcodebookscontaindescriptionsofthequestions,codes,andvariablesassociatedwithasurvey.Thecodebookmustbeassembledfortheanalysis.LookatthisexcerptfromthecodebookfortheCARPS.

Example:ExcerptFromtheCodeBookfortheCARPS,aSurveytoDetectBingeDrinking

Allvariablesarebrokendowntodiscreteunitscalledvalues,whichcorrespondtothecodesforthatvariable.Forinstance,thefrequencywithwhichapersonfeelsguiltyorsorryforsomethingheorshedidbecauseofalcoholusehasfivevalues:1=dailyoralmostdaily;2=atleastonceaweekbutlessthandaily;3=atleastonceamonthbutlessthanweekly;4=lessthanonceamonth;0=never;leaveblankifnodata.Thecodesare4,3,2,1,and0.Ifnoinformationfromtherespondentisavailable,nothingisfilledin.

Althoughstatisticalprogramsvaryinterminology,theyarefairlyconsistentinrequiringvariablestobenamedincapitalletters(PROJIDorGENDER)andtoavoidtheuseofspecialcharacters,suchascommasorsemicolons.Someprogramshavelimitsonthenumberofcharacters(usuallyabouteight)thatyoucanuse.Variablelabelsaretheactualnamesofthevariable(e.g.,“frequencyoffeelingguiltyorsorry”istheactualnameofthevariablenamedFGUILT).Tounderstandyourdata,thecomputerneedstoknowthenamesofthevariables(suchasFGUILT),thevariablelabels(frequencyoffeelingguiltyorsorry),andvaluelabels(dailyoralmostdaily=1;atleastonceaweekbutlessthandaily=2,etc.).(Pleasenote:Yourstatisticalprogrammayuseslightlydifferentterms,buttheideawillbeexactlythesame.)

Inlargesurveyprojects,thecodebookistheproject’sofficialrecord.Thecodebookcontainsvariablenames,labels,andvaluesorcodes;surveymethodsandcharacteristics;findings;andthesurveyinstrument(includingscoringsystem,ifrelevant)andsurveyteammembers.Ifyouoranyofyourcolleaguesandassociateseverwanttousethedatafromthesurvey(say,todoadditionalanalyses)or

repeatthesurvey(say,onadifferentsample),youwillfindthecodebooktobeessential.Becauseoftheirdetail,thistypeofcodebookissometimesreferredtoasanoperationsmanual.

Samplecodebooksintheirentiretycanbefoundontheweb.Tofindthem,gotoyourfavoritesearchengineandenter“surveysANDcodebook”or“+survey+codebook.”

EstablishingReliableCodingToensurereliabledatainsmallsurveys—say,withjustonepersondoingthecoding—youshouldrecodeallorasampleofthedatatocheckforconsistency.Thesecondcodingshouldtakeplaceaboutaweekafterthefirstcoding.Thisisenoughtimeforthecodertoforgetthefirstsetofcodessothattheyarenotjustautomaticallyreproduced.Afterthedataarecodedasecondtime,thetwosetsofcodesshouldbecompared.

Inlargeorverylargesurveys,asecondpersonshouldindependentlycodeasampleofthedata.Toensurereliabilitybetweencoders,providedefinitionsofalltermsandformallytrainallcoders.

Despiteyourbesteffortsatsettingupahigh-qualitycodebookanddatamanagementsystem,thecodersmaynotalwaysagreewithoneanother.Tofindoutabouttheextentoftheiragreement—intercoderorinterraterreliability—youcancalculateastatisticcalledthekappa.Kappameasureshowmuchbetterthanchancetheagreementisbetweenapairofcoders.Hereistheprinciplebehindthekappa.

MeasuringAgreement:TheKappaSupposetworeviewersareaskedtoindependentlyreview100interviewswithmembersofacommunitywhoareabouttovoteonwhethertheywantanewmalltobebuilt.Thereviewersaretostudythetranscriptsoftheinterviewstofindouthowmanypeoplementiondisplacementofsingle-ownerstoresbynationalchainsduringthediscussion.Thereviewersareaskedtocode1for“no”ifapersondoesnotmentiondisplacementatleastonceand2for“yes”ifheorshedoesmentiondisplacement.Thefollowingtableshowsthereviewers’codes.

Reviewer1saysthat30(A)ofthe100interviewsdonotcontainreferencetodisplacementofsingle-ownerstores,whereasReviewer2saysthat35(B)donot.Thetworeviewersagreethat20(C)studiesdonotincludementionofdisplacement.

Whatisthebestwaytodescribetheextentofagreementbetweenthereviewers?Twentyof100or20%(C)isprobablytoolowbecausethereviewersalsoagreethat55%(D)ofstudiesincludementionofdisplacement.Thetotalagreement(55%+20%)isanoverestimatebecause,withonlytwocategories(yesandno),someagreementmayoccurbychance.

ThisisshowninthefollowingformulainwhichOistheobservedagreementandCisthechanceagreement.

MeasuringAgreementBetweenTwoCoders:TheKappa(κ)Statisticκ=O−C(Agreementbeyondchance)1−C(Agreementpossiblebeyondchance)

Hereishowtheformulaworkswiththepreviousexample.

1. CalculatehowmanyinterviewsthereviewersmayagreethatbychanceDONOTincludementionofdisplacementofsmallstores.Thisisdonebymultiplyingthenumberofnoanswersanddividingby100becausethereare100interviews:30×35/100=10.5.

2. CalculatehowmanyinterviewstheymayagreethatbychanceDOincludementionofdisplacementbymultiplyingthenumberofinterviewsthateachreviewerfoundtoincludemention.Thisisdonebymultiplyingthenumberofyesanswersanddividingby100:70×65/100=40.5.

3. Addthetwonumbersobtainedinsteps1and2anddivideby100togetaproportionforchanceagreement:(10.5+45.5)/100=0.56.

Theobservedagreementis20%+55%=75%or0.75.Therefore,theagreementbeyondchanceis0.75−0.56=0.19,thenumerator.

Theagreementpossiblebeyondchanceis100%minusthechanceagreementof56%or1−0.56=0.44,thedenominator.k=0.190.44k=0.43

Whatisa“high”kappa?Someexpertshaveattachedthefollowingqualitativetermstokappas:0.00−0.20=slight;0.20−0.40=fair;0.40−0.60=moderate;0.60−0.80=substantial;and0.80−0.10=almostperfect.

Howdoyouachievesubstantialoralmostperfectagreement—reliability—amongreviewers?Youdothisbymakingcertainthatallreviewerscollectandrecorddataonexactlythesametopicsandthattheyagreeinadvanceonwhateachimportantvariablemeans.The“fair”kappaof0.43obtainedbythepreviousreviewerscanbeduetodifferencesbetweenthereviewers’andsurveyors’definitions,poortrainingintheuseofthedefinitions,andmistakesincoding.

ReviewingSurveysforMissingDataMissingdataresultfromunansweredquestionswithinasurveyorfailuretocompletethesurveyatall.Youshouldreviewthefirstcompletedsurveysassoonasyoureceivethem,beforeyouenteranydata.Inself-administered,writtensurveys,especiallymailedquestionnaires,expecttofindunansweredquestions.Respondentsmaynotanswerquestionsbecausetheydonotwantto,theymissthequestions(donotseethem),ortheydonotunderstandthem.Peoplemaynotunderstandthedirectionsforcompletingthesurveyorthequestionsbecausetheamountofreadingistoogreatorthereadinglevelistoohigh.Respondentsmaybeunsureofhowtorespond(completelyfillintheboxes,circlethecorrectanswer,etc.).Theymayfindtheformatdifficulttouse.

Confusingquestionformatsleadtomissingdatabecauserespondentsdonotknowhowtoanswerthe

question.Extensivecognitivepretestsandpilottestingmayovercomesomeproblemsthatcauserespondentstomisunderstandsurveyquestions.Cognitivepretestsareinterviewswithpotentialrespondentsthataskthemtointerpreteachquestionandresponse.Pilottestsaretestsofthesurveyintheplannedsurveysetting.Thesetwoactivitieswilltellyou,beforeyougotothefield,thataparticularformatisunusableorthatsomequestionsdonotmakesense.Pleasenotethatyoushouldpilottestallsurveytypes,includingcommerciallyproducedonlinesurveys.Thequestionformatmaybeinappropriateformanyrespondents,thesurveymaycontainseeminglyirrelevantquestions,thesitemaybedifficulttonavigate,andsoon.Itisaxiomatic,however,thatnomatterwhatyoudo,youcanexpectsomemissingdata:Thatisasurething.

Amajorproblemfacedbysurveyorsishowtohandlemissingdata.Sayyoumail100surveysandget95back.Youproudlyannouncethatyouhavea95%responserate.Aftercloserexamination,youdiscoverthathalftherespondentsdidnotanswerQuestion5andthatnot1ofthe25surveyquestionswascompletedbyeveryrespondent.Withallthatmissinginformation,youcannotreallyclaimtohavea95%responserateforallquestions.

Whatshouldbedoneaboutmissingresponses?Insomecases,youmaybeabletogobacktotherespondentsandaskthemtoanswerthequestionstheyomitted.Insmallsurveys,wheretherespondentsareknown—say,inanofficesettingorwithinaschool—therespondentsmaybeeasilycontacted.Butcollectinginformationasecondtimeisusuallyimpractical,ifnotimpossible,inmostsurveys.Somesurveysareanonymous,andyoudonotevenknowwhotherespondentsare.Ininstitutionalsettings,youmayhavetogobacktotheInstitutionalReviewBoardtogetpermissiontocontacttherespondentsasecondtime.Thisprocessmaytaketimeanddelayyoursurvey’scompletion.Onceagain,statisticalmethodsareavailablethatcanbeusedtoaccountformissingdata.Somearebasedonassumptionsabouthowtheaveragerespondentwouldhaveresponded.

Onlinesurveyscanbeprogrammedsothattherespondentmustansweronequestionbeforeproceedingtothenext.Somerespondentsmayfindthisapproachfrustrating,however,andrefusetocompletethesurvey.Althoughcompellingrespondentstoanswerallquestionsistoutedasamajoradvantageofonlineandothersurveys,somesurveyorsthinkthatforcingrespondentstoanswereveryquestioniscoerciveandunethical.Thesesurveyorsarguethat,inmanysurveys,participationisvoluntary;respondentscanquitwhentheywant.Asurveythatforcespeopletoansweraquestioneveniftheyprefernottomaybeconstruedbysomepeopleasviolatingtheethicalprincipleofautonomyorrespectforindividuals.Moreover,thesurveymayresultinunreliableinformationbecausesomepeoplemayenterameaninglessanswerjusttobeabletomoveontothenextquestion.

EnteringtheDataDataentryistheprocessofgettingthesurvey’sresponsestothestatisticalsoftwareinareadableformat.Fromthere,itgoestoadatabase.Itusuallytakesthreeforms.Inthefirst,datafromacodedsurveyareenteredintoadatabasemanagementprogramorspreadsheet.ThedataarethensavedintextorASCIIfilessotheycanbeexportedintoastatisticalprogramsuchasSPSS,SAS,orStata.AsecondtypeofdataentryinvolvesenteringdatadirectlyintoastatisticalprogramsuchasSPSS,SAS,orStata.Inthethirdformofdataentry,therespondentorinterviewerentersresponsesdirectlyintothecomputer.Dataentryofthistypeisassociatedwithcomputer-assistedinterviewing,onlinesurveys,andscannedsurveys.Theresponsesareautomaticallyenteredintodatabasemanagementsystemsorstatisticalprograms.Programsarealsoavailablethatwillautomaticallyconvertonefileformatintoanother(say,fromSAStoStata).

Eachdataentry,databasemanagement,andstatisticalprogramhasitsownconventionsandterminology.Someprogramswilltellyouthatwhenenteringdatayouaresettinguprecordsforeachrespondent.Therecordconsistsoftheperson’suniqueID(identificationcode)andtheperson’sobservations(responsechoices,scores,comments,etc.).Otherprogramsconsidertheunitofanalysis(suchastheperson)astheobservationandthedatacollectedoneachobservationasthevariablesorfields.Theexamplethatfollowsisasimpledatasetforsixpeople.

Example:SurveyResponsesFromSixPeople

Inthisexample,thetableisorganizedsothattherowsareforeachpersonandthecolumnsaretheirdata.Thatis,Person2’sdataare2,4,1,3,and2.RespIDisanabbreviationfor“respondentidentificationnumber.”Manystatisticalprogramsrequireyoutotellthecomputerwhereonthedatalineavariableislocated.Forinstance,intheexample,theperson’sgenderiscalledGENDER,anddataongendercanbefoundincolumn2.BingedrinkingiscalledBINGE,andthedataforthisvariablecanbefoundincolumn5.

Databasemanagementprograms,statisticalprograms,andcomputer-assistedsurveyswithautomaticdataentrycanfacilitateaccuracybybeingprogrammedtoallowtheentryofonlylegalcodes.Forinstance,ifthecodesshouldbeenteredas001−010,thenyoucanwriterulessothatanentryof01or10isnotpermitted.Ifyoutrytoenter01or10,youwillgetanerrormessage.Withminimumprogramming,theprogramcanalsocheckeachentrytoensurethatitisconsistentwithpreviouslyentereddataandthatskippatternsarerespected.Thatis,theprogramcanmakesurethatthefieldsforquestionsthataretobeskippedbysomerespondentsarecodedasskipsandnotasmissingdata.Designingacomputer-assistedprotocolrequiresskillandtime.Noprotocolshouldberegardedaserror-freeuntilithasbeentestedandretestedinthefield.

CleaningtheDataOncethedataareentered,theyneedtobecleaned.Acleandatasetcanbeusedbyanyonetogetthesameresultsyoudowhenyouruntheanalysis.Databecome“dirty”foranumberofreasons,includingmiscoding,incorrectdataentry,andmissingresponses.

Onegoodwaytoavoiddirtydataistorelyonscannedoronlinesurveys.Ifyouenterdatabyhand,however,makesurethatcodersordataenterersareexperienced,welltrained,andsupervised.

Runfrequenciesonyourdataassoonasyouhaveabout10%oftheresponsesin.Runthemagainand

againuntilyouaresurethatthesurveyisrunningsmoothly.Frequenciesaretabulationsoftheresponsestoeachsurveyquestion.Ifyourdatasetisrelativelysmall,youcanvisuallyscanthefrequenciesforerrors.Forlargedatabaseswithmanytablesorrecords,variables,skippatterns,andopen-endedtextresponses,youmayneedtodoasystematiccomputerizedcheck.Allleadingstatisticalprogramsprovideforcleaningspecificationsthatcanbeusedduringdataentryandlaterasaseparatecleaningprocess.

Severalotherproblemsmayrequireyoutocleanupthedata.Theseincludehavingtodealwiththecompleteabsenceofdatabecausesomesurveyshavenotbeenreturned,missingdatafromsurveysthathavebeenreturned,andsurveysthatcontaindatathatareverydifferentformtheaveragerespondents.

Tominimizelowresponserates,makesurethesurveyismeaningfultotheintendedresponders.Isthetopicimportantenoughsothattheyfeelcompelledtoanswerthequestions?Besuretoprovetotherespondentthatyouwillrespecttherulesofconfidentiality.Checkthesurvey’sreadabilitylevel,format,andaptnessofthelanguagethroughcognitivepretestingandscientifictranslation.Ifthereadinglevelistoohigh,theformatiscluttered,orthetranslationispoor,thenyoucancountonpeopletoignorethesurvey.Ifyouareconductinganinterview,spendthetimeandmoneytotrainandmonitorinterviewers.Youcanlosepeopleiftheydonotunderstandorlikethesurveyorinterviewer.Itoftenhelpstoofferfinancialorotherincentives(suchasgiftsorgiftcertificates)toencouragerespondentstorespond.

ValidatingSurveyDataOnlinesurveysoftwarealmostalwaysletsyouvalidatetheresponses.Forexample,ifaquestionasksforadate,youcandecideonhowthatdateisexpressed.Supposeyoursurveyasksforaperson’sbirthdate.Youcanprogramtheresponseoptionsothattherespondenthasnochoicebuttorecordthedataasmonth,day,andyear.Individualswhouseanothersystem,say,day,month,andyear,willnotbeallowedtorecordtheirresponse.Theywilleithergetamessagesuchas“invalidresponse,”theircomputerwillgiveawarningsound,orboth.Thevalidationfeaturealsoguaranteestheuniformityofresponsesthatisessentialtoquantitativeanalysis.

Accesstothevalidationfeatureusuallyrequiresthatyoupayforahigherlevelofthesurveysoftwarethanjustthebasic,sometimesfree,version.

Somevalidationinstructionsandoptionscommonlyfoundinonlinesurveysoftwareincludethefollowing:

don’tvalidatecommenttextmustbeaspecificlengthmustbeawholenumbermustbeadecimalnumbermustbeadate(MM/DD/YYYY)mustbeadate(DD/MM/YYYY)mustbeanemailaddress

SummingUpSurveydataanalysisusesstatisticalandqualitativemethodstodescribe,compare,andinterpret

respondents’attitudes,preferences,values,andbehavior.Whenchoosinganappropriateanalysismethodforyoursurvey,youmustconsideranumberoffactorssimultaneously.Thesefactorsarethesurvey’ssamplesizeandresearchdesign,aswellasthecharacteristicsandthequalityofthedata.Tochooseamethod,answerthesesevenquestions:

Howmanypeopleareyousurveying?Areyoulookingforrelationshipsorassociations?Willyoubecomparinggroups?Willyoursurveybeconductedonceorseveraltimes?Arethedatarecordedasnumbersandpercentagesorscoresandaverages?Howmanyindependentanddependentvariablesinterestyou?Arethedataofhighquality?Thatis,aretheyreliableandvalid?

Contentanalysisisusedtoanalyzequalitativedataforthepurposeofdrawinginferencesaboutthemeaningofrecordedinformationsuchastheopen-endedresponsesandcommentsmadebysurveyrespondents.Allsurveyreturnsshouldbereviewedformissingdataandaplanshouldbedevisedforhandlingthem.Missingdataresultfromunansweredquestionsorentiresurveys.Dataentryistheprocessofgettingthesurvey’sresponsesfromtherespondenttothestatisticalsoftwareinareadableformatsotheycanbeputintothesurvey’sdatabase.Onlinesurveydatacanbeautomaticallyenteredintoadatabase,withtheluxuryofreceivingreportsinrealtime.Surveydataalsomaybescannedinorhandentered.Regardlessofhowthedatagetintodatabases,theresponsesneedtobecheckedforaccuracy.Acleandatasetisonethatcanbeusedbyanyonetogetresultsthatareidenticaltoyours.Onlinesurveysoftwareusuallyhasavalidationoptionthathelpsstandardizedatacollectionandmakessurethattheinformationthatrespondentsprovideistheinformationaskedforbythesurvey.

ThinkAboutThis1. Findthethreearticleslistedbelow.Nameanddiscussthedata-analyticmethodsthatareusedto

interpretthesurveyresultsdiscussedineacharticle.YoucanlocatethearticlesthroughPubMed(http://www.ncbi.nlm.nih.gov/)throughanysearchenginebyenteringthenameofthejournal.

Article1.Mazzucato,V.,Cebotari,V.,Veale,A.,White,A.,Grassi,M.,&Vivet,J.(2014).Internationalparentalmigrationandthepsychologicalwell-beingofchildreninGhana,Nigeria,andAngola.SocialScience&Medicine,132(2),215–224.

Article2.O’Connor,T.M.,Yang,S.-J.,&Nicklas,T.A.(2006).Beverageintakeamongpreschoolchildrenanditseffectonweightstatus.Pediatrics,118(4),e1010–e1018.

Article3.Richardson,J.,Coid,J.,Petruckevitch,A.,Chung,W.S.,Moorey,S.,&Feder,G.(2002).Identifyingdomesticviolence:Crosssectionalstudyinprimarycare.BritishMedicalJournal,324(7332),274–278.

2. AnswerthesequestionsabouteachofthetwosurveyslistedinQuestion1(oranyothers):Howmanypeopleweresurveyed?Didthesurveyinvestigatorsdeterminerelationshipsormakepredictions?

http://www.ncbi.nlm.nih.gov/

Didtheinvestigatorscomparegroups?Wasthesurveytheydiscussedconductedonceorseveraltimes?Werethedatanominal?Continuous?Both?Howmanyindependentanddependentvariablesdidthesurveyaddress?Werethedataofhighquality?Thatis,didthesurveyorsdemonstratethattheywerereliableandvalid?

7PRESENTINGTHESURVEYRESULTS

OverviewSurveyresultscanbeshownonthesurveyformandintables,graphs,diagrams,orpictures.Surveyreportsoftenincludesummarytables.Theirpurposeistodescriberespondentsandshowrelationshipsandchanges.Followtherulesifyouplantousetables.Forexample,usethetable’scolumnstoshowtheindependentvariables,suchasgroupandtimingofthesurvey(e.g.,baseline,6monthslater,and3yearslater).Selectatableformatanduseitconsistently,relyonverticallinessparingly,andpresentdatainalogicalorder,suchasfrommosttoleastfrequent.

Piediagramsshowtheproportionoccupiedbyeachresponsecategory.Eachsliceofthepiemustbeequalinproportiontothenumberorpercentageofresponsesitrepresents.

Bargraphsarecommonlyusedtodisplaysurveydatabecausetheyprovideanoverviewofseveralkindsofinformation—comparisonsandchanges—atoneglance.Linegraphsalsoshowchangesanddifferencesingroups.Becarefulnottooversellandmakeaone-pointchangeinscorelooksignificantunlessitreallyis.

Manysurveysresultinwrittenreports.Considerincludingthesesectionsinyours:abstract,summary,tableofcontents,figures,glossaryofterms,statementofpurposesorobjectives,methods,results,conclusions,recommendations,references,index,appendix,andacknowledgments.

Slidepresentationsmaybethebestchoiceifyouplantomakelast-minutechangesandifyouwantimmediatefeedbackfromtheaudience.Anotheroptionistheposter,whichprovidesreaderswithacondensedandvisualexplanationofthesurvey’sobjectives,methods,results,andfindings.

ReproducingtheQuestionnaireSurveyresultscanbeshownonthesurveyformorintables,graphs,diagrams,orpictures.Lookatthis.

Example:ReportingResultsWiththeQuestionnaireSource:SurveyMonkey®

Inthepast4weeks,howmuchofthetimehaveyoufeltdepressed?

Theadvantageofusingthesurveyformtogiveresultsistoletthereaderorlistenerseethequestionsandresponsechoices.Thisapproachshowsthesurvey’srawdata.Itprovidesthedatawithoutanyanalysisorinterpretation.

Onlinesurveysystemsprovideinformationonareal-timebasis.Asresponsescomein,thesystemautomaticallygivesyouinformation,suchasthenumberofpeoplewhocompletedthesurveysofarandhowtheresponsesaddup.HereisanexamplefromQuestionPro.

Example:OnlineSurveyStatisticsinRealTime

UsingTablesSurveyreportsnearlyalwayshavetables.Theirpurposeistodescriberespondentsandshowrelationshipsandchanges.Lookattheseshelltables.

Example:ShellTableDescribingChildreninTwoSchools

Example:ShellTableforComparingChildreninTwoSchools

Thefirsttableprovidestheformatforlistingthepercentageofchildrenintwoschoolswhohavecertaincharacteristics(suchasageandgradepointaverage).Thesecondtableletsyouputinstatisticsusedtotestwhetherthedifferencesaremeaningful.

Table7.1isanotherexamplethatshowsstatisticallysignificantdifferences(indicatedbyoneandtwoasterisks).Thistablealsohasasourcelisted:TheNationalChildren’sHealthSurvey,CenterforHealthStatistics.Includethesourceoftheinformationwhenitisnotobvious.Thesourceisnotobviousifitcomesfromanyonebutthereporter.Notice,too,thatthescoresareexplainedinthetablefootnote.

Source:Self-administeredsurveyadministeredbyTheNationalChildren’sHealthSurvey,CenterforHealthStatistics.

aScoresof75to100indicatehealthylifestyles.Scoresof26to74indicateneitherhealthynorunhealthylifestyles.Scoresof1to25indicateunhealthylifestyles.

*p=.003betweenhealthyandunhealthywithinschool;

**p=.002betweenhealthyandunhealthywithinschool.

LookatTables7.2and7.3.

Whichtabletellsyoumore,Table7.2orTable7.3?Iftheemphasisofthesurveyisonchangesbetweengroups(youngerthan65yearsvs.65yearsandolder),thenTable7.2isbetter.Ifyouwanttoemphasizethechangesthattookplaceovera1-yearperiod,thenTable7.3isbetter.

Source:TheSelf-EfficacyScale(NewYork:NationalPress)Note:Scaleis1to9,withscoresof1to3=high;scoresof7to9=low.

Source:TheSelf-EfficacyScale(NewYork:NationalPress).Note:Scaleis1to9,withscoresof1to3=high;scoresof7to9=low.

SomeTablePreparationRules

1. Tablesdisplaycolumnsandrowsofnumbers,percentages,scores,andstatisticaltestresults.Decidehowmanycolumnsandrowsyoucanincludeandstillkeepthetablereadable.

2. Eachtableshouldhaveatitlethatsummarizesitspurposeandcontent.3. Whenthesourceofatable’sdataisnotimmediatelyobvious,itshouldbegiven.4. Whenyouuseatermthatmaybeconfusing,defineit.Setoffdefinitionswithasterisksor

superscripts.5. Columnsaretheindependentvariables,suchasgroupandtimingofthesurvey(e.g.,baseline,6

monthslater,and3yearslater).6. Selectatableformatanduseitconsistently.Thetablesinthepreviousexamplesusecaptionsin

whichthefirstletterofeachword(exceptprepositionsandarticles)inthecolumnsiscapitalized.Intherows,onlythefirstwordiscapitalized.Manypeopleuseonlyhorizontallinesintables.Useverticallinessparingly,especiallyifyoualsousehorizontallines.

7. Presentdatainsomelogicalorder.Onecommonlyusedorderisfrommostfrequenttoleastfrequent,althoughthereversemaybeappropriate,too.Theideaistobelogicalsothatthereadercanfollow.

8. Includethesamplesizeanddifferentiateamongnumbers,percentages,andotherstatistics.

Allwordprocessingprogramscanhelpyouformattablessotheyareeasytoread.Ifyouusetablesfrequently,learnhowtouseyourprogram’stablefunctions.

DrawingPieDiagramsPiediagramsshowvisuallytheproportionofthewholethateachresponsecategoryoccupies.Supposeyouwereconductingasurveyof80libraryusers’needsandwantedtodistinguishtheneedsofpeopleofdifferentages.Supposealsothatyoufoundthat40respondentswererelativelyyoung—say,between18and25yearsofage—andonly10werebetween25and35,withtheremaining30peopleolderthan45.YoucandescribeyourfindingseffectivelybypresentingthemasshowninFigure7.1.

Figure7.1LibraryUsers:N=80

Figure7.2LibraryUsers:N=80

LookatFigure7.2.Thispiecharthasthesameproportionofusersaspreviously.Bycuttingthepie,youemphasizethedifferentsizesofeachslice.

Ifyouplantousepiediagramsfrequently,youcanrelyongraphicsprograms.Spreadsheet,wordprocessing,andotherprogramsincludegraphicsfunctionsthatenableyoutodrawpiesfromtablesofdata.Thekeytoanaccuratepiediagramisitsscale.Eachslicemustbeequalinproportiontothenumberorpercentageofresponsesitrepresents.Thatis,50%ofresponsesishalfthepie,25%isonequarter,andsoon.Remembertokeeptheslicestonomorethanaboutsixorthepiewillbetoocluttered.

UsingBarGraphsBargraphsarecommonlyusedtodisplaysurveydatabecausetheyprovideanoverviewofseveralkindsofinformationatoneglance.LookatFigure7.3,agraphofchangesinbehaviorbetweenboysandgirlsfrom1985to2005.Thisgraphtellsyoutwothingsatonce:

1. Boys’andgirls’behaviorchangedovertime.2. Boyshadconsistentlyhigherscoresthandidgirls.

NowlookatFigure7.4.Thisgraphofthesameinformationconcentratesonboysandgirls,althoughyoustillgettwokindsofinformationatoneglance.

Figure7.3Boys’andGirls’BehaviorSurvey


Figure7.5isagraphofthesameinformationfocusingonscores,andyoucanseetwokindsofdataatonce.

Bargraphsshouldalwayshaveatitle,alegendorkeytothebars,andanyotherexplanationsneededtokeeptheresultshonest.Rememberthatseeingcanbedeceiving.Justbecausethegraphsuggestsdifferencesdoesnotmeantheyarereal(significantandpractical).

Figure7.6showsthesamedatapresentedinalinegraph.Linegraphsarebetterthanbarsatshowingtheflowofchangeovertime.Mostgraphicsprogramswillallowyoutoautomaticallyswitchfromonetypeofgraphtoanothersothatyoucanactuallyseewhichonebestdescribesyoursurveydata.Theobjectiveisaccuracy.

UsingLineGraphs

Linegraphsaredrawingsthatallowyoutoshowchangesandcomparegroups.Becarefulnottooversellandmakeaone-pointchangeinascorelooksignificantunlessitreallyis.Lookatthesescores:


Figures7.7and7.8aretwographs,eachofwhichhasbeenplottedasalinegraphtorepresentthescoresandtheirfrequency(number).Whichgraphismoreaccurate?Figure7.7isthecomputer’schoice.Figure7.8isnot.Inthesecondgraph,thedifferenceinscorescertainlylookslessimpressive.


Figure7.7

Whichispreferred,abaroralinegraph?Ifindoubt,experiment.Toshowcomparisons,tablesandbargraphsseemtoworkbest.Forchangesovertime,linegraphswin.

Onlinesurveysystemshavetheadvantageoflettingyouquicklyexplorethebestwaytopresenttheresults.Thefollowinggivesapartialexampleoftheresultsofacompanysupportsatisfactionsurvey.Followingthatisanillustrationofanonlinesurveysystem’sreportofacustomersatisfactionsurveyandabarandlinegraphdepictingtheresults.

Figure7.8

Thesurveyorcreatedthegraphsbytransferringtheresponsesfromthesurveytoaspreadsheetprogramandusingits“wizard”function.Thebargraphisusefulinseeinghoweachratingchangedovertime.Thelinegraphisbetteratshowingmovement.Itshouldbenotedthatthegraphsareeasiertounderstandincoloronthescreenorwhenprinted.

Example:DownloadedResultsofaCustomerSatisfactionSurveySource:Takenfromwww.QuestionPro.com.

http://www.QuestionPro.com

Example:TransferringtheResultsofOneQuestionIntoBarandLineGraphs

DrawingDiagramsorPicturesUsepicturesordiagramstogetyourpointacross.Supposeyouwanttodescribeyoursurvey’sresearchdesign.Comparethewordswiththediagram.

Allmajorwordprocessingprogramsandspreadsheetshavefunctionsthatallowyoutoeasilyconvertatableofdataintoachart.Ifyouuseflowchartsoften,youshouldconsiderbuyingagraphicsprogram.

Example:WordsandDiagramsinSurveyReportsWeusedadesigninwhichschoolswerethesamplingunits.Fourteenschoolswith2,000totalstudentswereeligibletoparticipateinaninjurypreventionprogram.Werandomlyselected10schoolswithatotalof945students.Fiveschoolswithatotalof443studentswererandomlyassignedtoreceivethecurriculum.Ofthefiveschoolsthatreceivedthecurriculum,425studentscompletedthebaselinesurvey.Ofthe502studentsnotreceivingthecurriculum,450completedthebaselinesurvey.Weobtained12-monthfollow-updataon400studentsinthefiveschoolsreceivingthecurriculum.Oneschoolnotreceivingthecurriculumdroppedout,andwecollected12-monthfollow-upsurveydataintheremainingfourschoolson213students.

Nowlookatthisdiagram.

Adiagramlikethisishelpfulinexplainingcomplexresearchdesigns.Thediagramreliesontheprinciplesusedtomakeanorganizationalorflowchart.

WritingtheResultsofaSurveyAfairlytypicalconclusiontoasurvey’sactivitiesisawrittenreportofitspurposes,methods,results,conclusions,andrecommendations.Onequestionalwaysseemstocomeupwhenyousitdowntoorganizethereport:Howdetailedandtechnicalshouldyoube?Ifyouaretootechnical,youmayreduceyourreadershipsubstantially;however,ignoringthetechnicaldetailsmaysubjectyourreporttocriticism.

Threeaimsshouldguideyourwriting:

1. Becomprehensive.2. Organizecarefully.3. Writeasclearlyasyoucan.

Soundlikeoneofyourearliestschoolteacherswarningyouaboutabookreport?Perhaps.Butifthesethreecriteriaareapplied,yourreportismorelikelytobereadthanotherwise.

Achievingcomprehensivenessmeansincludingasmuchaspossiblesothatanyonewhowantstocanunderstandthereport.Youmustorganizethereportsothatanyonecanfindoutwhatheorshewantstoknow.IfIaminterestedinwhyyouchoosetheXYZSelf-EfficacySurvey,Ishouldbeabletofindthatinformationeasily.Writingclearlyisalwaysadesirableaim.Insurveyreports,claritymeansusingstandardlanguageconventionsandmakingsurethatallambiguoustermsaredefined.

OrganizingtheReportConsiderincludingallthese:

AbstractSummaryTableofcontentsListoftablesandfiguresGlossaryoftermsRoadmap:WhattolookforinthereportandwheretofinditStatementofpurposesorobjectivesOutcomesMethodsResultsConclusionsRecommendationsReferencesIndexAppendixAcknowledgments

Abstract

Theabstractisusuallyabout150to250words.Considerusingastructuredabstract.Lookatthis.

Thisabstracthas152words,countingtheheadings.Yourwordprocessingprogramislikelytohaveafunctionforcountingwords.

Example:StructuredAbstractofaSurveyReportPurpose:ToidentifycommunityattitudestowardintegratingboysandmenintoWomenForward(WF),atraditionallyall-femalegrassrootsorganizationaimedatencouraginggirlsandwomentoenterpolitics

Researchdesign:Across-sectionalsurveyof150people

Survey:Fifteen-minutetelephoneinterviewsconductedduringJuly2004

Respondents:An82%responseratewasobtained.Oftherespondents,70%weremen,and62%ofrespondentswere35yearsofageoryounger.Respondentswererandomlyselectedfromlocalpoliticalorganizations.

Mainoutcome:Supportforintegration

Results:Eighty-fourmen(97.6%)and32women(86%)supportedintegration.Thesedifferencesarenotsignificant.Respondentswhoare35yearsofageandyoungeraresignificantlylesseagerthanolderrespondents(p=.04)tointegratemenintoWF.

Conclusions:AlthoughmenandwomenareaboutequalintheirsupportofintegratingmenintoWF,youngerrespondents—regardlessofgender—arelessenthusiastic.Interpretthesefindingscautiouslybecause70%ofrespondentsweremale.

Summary

Thesummaryisadistillationofallthereport’skeycomponents(objectives,methods,findings).Itshouldbeaboutthreepageslongandtake5minutestoread.Youcanprovidemoredetailthanyoudointheabstractandaddinformationonwhoconductedthesurvey.Youcanalsoincludeoneortwoimportanttablesorfigures.

Tableofcontents

Thetableofcontentsshouldlistallmajorsectionsofthereportwiththeassociatedpagenumbers.

Listoftablesandfigures

Listeachtableandfigureandgiveitscompletenameandthepagenumberonwhichitappears.Insomereports,tablesandfiguresgoattheendinaseparatesection.

Glossaryofterms

Alltechnicalterms(e.g.,randomsampling),abbreviations(e.g.,CBS=CenterforBusinessandScience),andambiguousconcepts(e.g.,healthy,smart,hostile)canbeincludedintheglossary.

Roadmap:Whattolookforinthereportandwheretofindit

Somereportsarecomplex,andthereaderneedsaroadmaptofollowthem.Forinstance,supposeyourreporthastwomajorcomponents:Part1andPart2.Tellthereaderhowtofindeachpart(Separatelybound?Clickonaword?Anicon?).Tellwhatisineach(e.g.,Part1hasadescriptionofthesurveyproject;Part2discussesthevalidation).

Statementofpurposesorobjectives

Thepurposesarethesurvey’sobjectives.Discussinthereportwhyyoudidthesurvey.Didyoudothesurveytofindouthowsatisfiedparticipantsarewiththeirbenefits?Tofindoutwhichbenefitswillsatisfythem?

Outcomes

Onwhichmaindependentvariablesoroutcomesdoesthesurveyfocus?Thesemaybechangesinknowledge,attitudes,behavior,health,orqualityoflife.

Methods

Whatwasdone?Withwhom?Wastheeffortworthwhile?Toanswerthesequestions,discussthefollowing:

Thetypeofsurvey(awritten,self-administeredquestionnaire,telephoneinterview,etc.)andthelimitationsthatresultedbecauseyouchosethattypeofsurvey.Forexample,supposethatinatelephonesurveyoftheteensinCountyX,youusetheirhomelandlinestoreachthem.Youuselandlinesbecauseyoucanconvenientlygetthosenumbersfrompublicrecords.However,almostallteensinCountyXaremorecomfortableusingtheirpersonalmobilephones.Ifyoupersistinusinglandlines,howwillthataffectthesurvey’scredibility?Surveyquestionsasked:Giveexamples;includethesurveyintheappendixifyoucannotincludeitinthetext.Surveylogistics:Duringwhichmonths,overwhatperiod,andhowfrequentlywasthesurveyadministered?Bywhomwasitadministered?Howweretheadministratorstrained?Howwasthequalityoftheadministrationmonitored?Whatarethelimitationsthatresulted?Forexample,sayyoutrainedfivepeople,butonewasnotadequate.Whatmightthishavedonetotheresults?Surveyconstruction:Givetheoriginofquestions,describepilottesting,anddiscussreliabilityandvalidity.Discussthelimitationsthatresultedduringconstruction.Forexample,thevalidationstudymayrevealthatthesurveyisnotusefulwithpeoplewhodonotreadEnglishorSpanishattheninth-gradelevel.Samplingandresponserate:Whatweretheeligibilitycriteria?Howdidyoucalculatetheresponserate?Howadequatewastheresponserate?Didyouuseincentives?Discussthelimitations.Becauseyoudidnotgetsufficientnumbersofeligiblerespondents,whatdidthisdototheresults?Forexample,supposeyouwantedhalfmenandhalfwomen,butrespondentswereprimarilyfemale?Surveyresearchdesignanditslimitations.Forexample,supposeyoursurveyusedcomparisongroupsthatwerenotrandomlyconstituted;whattypesofselectionbiascanyouanticipate?Howseriouslydoesthisaffectyourfindings?Analysis:Foreachmainsurveyoutcomeofinterest,describetheanalysismethod.Ethics:Wererespondentsproperlyinformedoftheirrightsandresponsibilities?Howwasconfidentialitymaintained?

Results

Concerningeachoutcomeofinterest,whatdidyoufind?Usuallythesearegivenasstatistics.

Conclusions

Whatdothefindingsmean?

Recommendations

Whatdoyourecommendbasedonthedata?Warning:Noteverysituationwarrantsrecommendations.Somepeoplewhocommissionsurveysjustrequirethedata;theywillmaketherecommendations.

References

Citethoseyouhaveusedinthetext.Putthereferencesinalphabeticalorderorlistthemintheorderinwhichtheyappearinyourtext.

Index

Includeanindexonlyinlongreports.

Appendix

Includeanyrelevantmaterialthatistoocumbersometobecontainedinthereport.Thismaymeanputtingalongsurveyintheappendixratherthaninthebodyofthereport.Asaruleofthumb,15questionsorfewercanbeincludedinthereport.Keepinmindthatsomesinglequestionscanhavemanychoices,andthesemighthavetobeputintheappendix.Forexample,“Inthepast12months,haveyouhadanyoneofthefollowinghealthproblems:headaches,stomachaches,sprainedankleorwrist,cold?”Eachchoiceissimilartoasinglequestion:“Inthepast12months,haveyouhadheadaches?”“Haveyouhadstomachaches?”“Haveyouhadasprainedankleorwrist?”“Haveyouhadacold?”

Acknowledgments

Thanktheindividualswhoprovidedanyguidanceinthedesignofthesurveyoritsinterpretation.Amongthepersonstoconsiderarethesurvey’sparticipants,statistician,anddatacollectors.

SurveyReportingChecklistsandGuidesNeedsomehelpindecidingexactlywhattoincludeinyourreport?Guidanceintheformofchecklistsisavailable.Forexample,considertheChecklistforReportingResultsofInternetE-Surveys(CHERRIES),whichcanbefoundat:http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1550605/table/table1.CHERRIESrequiresthatinternetsurveyreportsgivedetailedinformationonthesurvey’sresearchdesign,IRBprocess,developmentandpretesting,recruitmentandadministration,andcalculationofresponserates.CHERRIESalsocallsforinformationonhowmultipleentriesfromthesameindividualwererecognized,andrequiresthesurveyornotonlytoexplainhowdatawereanalyzedbutalsotodescribehowincompleteresponseswerehandledintheanalysis.Hereisanexampleofhowthechecklistrequiresreportingofthesurvey’sresponserate:

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1550605/table/table1

GototheAmericanAssociationforPublicResearch’swebsite(http://www.aapor.org)forhelpwithdefinitionsandpointersformakingsureyouhaveacompleteandtransparentsurveyreport.

Transparentsurveyreportingandsurveyethicsgohandinhand.AstheCouncilofAmericanSurveyResearchOrganizations(CASRO)advises,surveyorsareobligatedtoinsurethatthefindingstheyreleaseareanaccurateportrayalofthesurveydataanddemonstratethattheyhaveputinplacecarefulchecksontheaccuracyofallstatistics.CASROrecommendsthatasaminimum,anygeneralpublicreleaseofsurveyfindingsshouldincludethesurvey’ssponsorshipaswellasadescriptionofthesurvey’spurposes,sample,datesofdatacollection,namesofthesurveyors,exactquestionwording,andanyotherinformationthatalaypersonwouldneedtomakeareasonableassessmentofthereportedfindings.

Clear-WritingTipsHerearesometipsonhowtowriteclearly.

1.Usetheactivevoicewheneverpossible.

Poor:Thereportisrelativelysimpleandisobviouslywrittenforthenonexpert,forthereareveryfewstatisticaltablesgiven.(Twentywords,threeareformsoftheverbtobe.)Better:Therelativelysimplereportisobviouslywrittenforthenonexpert,foritgivesveryfewstatisticaltables.(Seventeenwords,twoverbs)

Avoidthereisandthereare.

Poor:Thereareveryfewstatisticaltablesinthereport.Better:Thereporthasveryfewstatisticaltables.

2.Donotsprinklesentenceswithprepositionalphrases.Whenpossible,avoid,“inordertogiveareason”andreplaceitwith“togiveareason.”Convertaprepositionalphrasetoaparticiple:“Intheefforttogetreliableattitudemeasures”canbecome“tryingtogetreliableattitudemeasures.”Convertaprepositionalphrasetoanadjective:“Itisaquestionofimportance”canbecome“Itisanimportantquestion.”

http://www.aapor.org

Nearlyallwordprocessingprogramshaveagrammarfunction.Theyareexcellentinpointingoutpassivesentences.Theyalsoprovidealternativesandcanremindyouofthegrammaryoumayhaveforgotten.However,theyarenotperfect,andtheyarenotinterestedinstyle.

3.Tryareadabilityformula.Readabilitytestingpredictsthegradelevelofwrittenmaterial.Manypeoplefeelcomfortablereadingbelowgradelevel,butpeoplewillignorematerialthatisaboveit.Allmajorwordprocessingprogramswillgiveyoureadabilitystatisticsforsomeorallofyourreport.

Thisisasimpleformulathatgivesyouagoodideaofhowtoestimatereadability.TheresultofusingtheformulaiscalledtheFOGindex:

1. Takea100-wordsampleofyoursurveyreport.2. Computetheaveragenumberofwordspersentence.Ifthefinalsentenceinyoursamplerunsbeyond

100words,usethetotalnumberofwordsattheendofthatsentencetocomputetheaverage.3. Countthenumberofwordswithmorethantwosyllablesinthe100-wordsample.Donotcount

propernounsorthree-syllablewordformsendingin-edor-es.4. Addtheaveragenumberofwordspersentencetothenumberofwordscontainingmorethantwo

syllablesandmultiplythesumby0.4.

Supposea100-wordpassagecontainsanaverageof20wordspersentenceand10wordsofmorethantwosyllables.Thesumofthesewordsis30.Multiplying30by0.4givesaFOGindexof12thgrade.

Lookatthestructuredabstractfoundearlierinthischapter.Thereadabilitystatisticssuggestthattheabstractisofaveragedifficultyrequiringareadinglevelfrom8thto10thgrade.Mostreadabilityformulasonlylookatwords;theydonotincludenumbers.Ifyouaretalkingtoalayaudience,keepnumberssimple.Use76%or76.2%,ratherthan76.17%.

TheOralPresentationOralpresentationsfollowmanyoftherulesofwrittenreports.Speakclearlyandslowly,andpaceyourpresentationtotheaudience’sneeds.Youmustbeselectiveinoralreporting—often,adifficulttask.Makesureyougiveaudiencesusefulresults.Policymakerswanttoknowthebottomlinewhiletheoristswanttoknowaboutvalidity.Layaudiencesareprimarilyconcernedwithfindingsandparticipants.Ifyouhavetime,youcantellthemaboutthesurvey’smethodsandlimits.Allaudiencesexpectexamplesofthesurveyquestions.

SlidePresentationsUsethefollowingguidelinestoprepareslides.

OralVersusWrittenReports:ADifferenceinConversationOralreportsandwrittenreportsaredifferentinatleastoneimportantway.Oralreportsdependonyourabilitytospeakthedetails.Writtenreportsrelyonproseandpicturestospeakforyou.Comparethesetwo.

Example:TableUsedinaWrittenandanOralReport

WrittenReport

WrittenInterpretationoftheTableThetablegivesthebaselineandfollow-upmeansandtheobservednetdifferencesinscoresforfoursurveyoutcomes.Weusedanindependentttesttocomparechangesinmeanscoresfrombaselinetofollow-upfortheexperimentalandthecontrolgroups.Significantdifferenceswerefoundfavoringtheexperimentalstudentsinmoralvalues,religiousbeliefs,andsocialresponsibility.Differencesinsocialresponsibilityapproachedsignificancebutdidnotreachthep≤0.05,thelevelofsignificancechosenforthettest.

OralInterpretationoftheTableEventhetablelooksdifferent.Ithasbeensimplified(seepage190).

Thetablecomparesexperimentalandcontrolschools.(Pointtotheappropriatecolumns.)Weusedanindependentttesttocomparethedifferencesinmeanscoresfrombaselinetofollow-up.(Pointtotheappropriaterows.)Wefoundstatisticallysignificantdifferencesfavoringstudentsinexperimentalschoolsinmoralvalues,religiousbeliefs,andsocialresponsibility.Differencesinethicalbehaviorapproachedsignificancebutdidnotreachthep≤.05,thelevelofsignificancechosenforthettest.

Source:www.apua.org.

http://www.apua.org

PostersPostersareanimportantwaytopresentsurveyresults.Aposterisasummaryofyourworkthatisdesignedtobereadandunderstoodwithoutoralexplanation.Usingarelativelylargeprintout,yourelyonshorttextsectionsandgivetheresultsasbulletpoints.Mostsurveypostersalsoincorporategraphicalelementstohelpillustratekeypointsoftheresearch.Asampleposterisonpp.188–189.Inreallifeitis36inches(91.44cm)by48inches(121.92cm).

Apostershouldpresentanoverviewofthestudyincludingadescriptionofthesurveyorsurveys.It’snot

athesisorjournalarticle,sodon’ttrytocramallthedetailsontotheposter.Acasualviewershouldbeabletogetyourmessagein3to5minutesandreadallthetextinnomorethan10minutes.

Includethestandardsectionsofajournalarticle.Theseincludethefollowing:introductionandstudyobjectives,survey,methods(includingresearchdesign,sampling,anddataanalysis),results,andconclusions.Youmayalsowanttoincludeanabstract,acknowledgments,andreferences.

Apostershouldhaveamaintitlethat’sreadablefrom25feetaway.Peoplewillbewanderingthroughthepostersession,soyouneedtocatchtheireyefromadistance.Ageneralruleistousea72-pointtypeandacommonfontsuchasTimesNewRomanorArialforyourpostertitleandtouseasmallersizeofthesamefontforthesectiontitles.Useasimplecolorscheme.Don’tdistractpeoplebyusingtoomanydifferentcolors,fonts,andfontsizes.

Thinkabouthowyouplantoprintyourposterbeforeyoudesignit.Becausenoteveryprintingoptionoffersthesamepaperdimensionsandbecauselargerpostersizesgenerallycostmoretoprint,firstchoosethepapersizeforprintingandthendesignyourposteraccordingly.Checkwithyourprintingvendortofindoutwhetheryoushouldbeawareofanyspecificlimitationsorguidelines.

MicrosoftPowerPointisarelativelyeasy-to-usetoolforcreatingposters.AdobeIllustratorhasmorefeaturesandcanprovideveryprofessionalresults,especiallyforpostersincludinglotsofhigh-resolutionimages,butitismorecomplexandexpensive.Withalittlesearching,youcanfindfree(andofcoursethoseyoupayfor)onlinetemplatesforsettingupyourposter.Forinstance,oneonlinecompanyhasacollectionoffreePowerPoint(.pptand.pptxnativeformats)researchpostertemplates.YoudownloadtheappropriatePowerPointpostertemplate,addtext,images,andgraphicsandsenditbacktothecompanyforprinting.

SUMMINGUPTheadvantageofusingthesurveyformtogiveresultsistoletthereaderorlistenerseethequestionsandresponsechoices.Onlinesurveysystemsprovideinformationonareal-timebasis.Eachtableofsurveystatisticsshouldhaveatitlethatsummarizesitspurposeandcontent.Whenthesourceofatable’sdataisnotimmediatelyobvious,itshouldbegiven.Whenyouuseaterminatablethatmaybeconfusing,defineit.Setoffdefinitionswithasterisksorsuperscriptletters.Tablecolumnsaretheindependentvariablessuchasgroupandtimingofthesurvey(e.g.,baseline,6monthslater,and3yearslater).Piediagramsshowtheproportionofthewholethateachresponsecategoryoccupies.Bargraphsarecommonlyusedtodisplaysurveydatabecausetheyprovideanoverviewofseveralkindsofinformationatoneglance.Linegraphsaredrawingsthatallowyoutoshowchangesandcomparegroups.Usetheactivevoicewheneverpossibleinsurveyreports.Donotsprinklesentenceswithprepositionalphrases.Tryareadabilityformulaonyourreport’scontent.Limiteachslidetoonemainconcept.Allowthelistener1to2minutesperslide.Toemphasizepointsinaslidepresentation,underlinetitles,usebulletsorcheckmarks,numbereachpoint,orusecontrastingcolorstoseparatepoints.

Ingeneral,uppercaseandlowercaselettersareeasiertoreadthanalluppercase.Postersareawaytopresentasummaryofasurveysothatitisvisuallyaccessible.Postershavethesamecontentasresearchpapers,buttheypresentitsothatthereadergetsthemessageinunder10minutes.

THINKABOUTTHIS1. Prepareaslidepresentationforthecontentsofthischapter.2. Writeaparagraphdescribingthesurveyresultsshowninthetableandlinegraphbelow.

LineGraph:

3.Describehowyouwouldpresenttheresultsofthetableandlinegraphinanoralpresentation.Inwhatwayswouldthewrittenandoralpresentationsbesimilar?Different?

REFERENCESBurns,K.E.,Duffett,M.,Kho,M.E.,Meade,M.E.,Adhikari,N.K.,etal.(2008).Aguideforthedesignandconductofself-administeredsurveysofclinicians.CMAJ,179,245–252.

Draugalis,J.R.,Coons,S.J.,&Plaza,C.M.(2008).Bestpracticesforsurveyresearchreports:Asynopsisforauthorsandreviewers.AmJPharmEduc,72,11.

Eysenbach,G.(2004).ImprovingthequalityofWebsurveys:TheChecklistforReportingResultsofInternetE-Surveys(CHERRIES).JMedInternetRes,6,e34.

Huston,P.(1996).Reportingonsurveys:Informationforauthorsandpeerreviewers.CMAJ,154,1695–1704.

Kelley,K.,Clark,B.,Brown,V.,&Sitzia,J.(2003).Goodpracticeintheconductandreportingofsurveyresearch.IntJQualHealthCare,15,261–266.

Index

Abstractinsurveyreport,180–181Acknowledgmentsinsurveyreport,182Adapting/selectingsurveys,79–87Additivescales,60–62Administrationofsurveys

interviews,74–75self-administered,1,72–73

Aesthetics,survey,71AgencyforHealthcareResearchandQuality(AHRQ),84Alphavalue,106AmericanAssociationforPublicOpinionResearch(AAPOR),29AmericanAssociationforPublicResearch,183Angold,A.,130ANOVA,148,150,152,154Appendixinsurveyreport,182Arnold,R.,132Audiotape,asinformationsource,5Audiovisualaids,76Averages,138–141

Bargraphs,172–174,177Bias,11,47–48

sampling,99–100Bimodaldistribution,140,141Bloom,P.,129Branchingquestion,71–72Bricker,J.,109

Campbell,D.,127Casecontrol(historical)design,126Categoricaldata,155–156Categoricalscales,49Cellphonesurveys,14CentersforDiseaseControlandPrevention(CDC),22–23Checklist,53–54ChecklistforReportingResultsofInternetE-Surveys(CHERRIES),182–183Childrenandsurveys,54–57

ethics,26–28Children’sOnlinePrivacyProtectionAct(COPPA),26TheChildSelf-ReportofDelinquency,130Chi-square,149Chronbach’salpha,77Chuang,L.-H.,109

Cleaningdata,163–164Closedquestions,38,39–40

responsesfor,48–49writing,38,44–48

Clustersampling,simplerandom,98–99Codebook(operationsmanual),157–159CodeofFederalRegulations,19Coding

establishingreliable,159–161intercoderagreement,159–161LB/LLdata,40–44

Coefficientalpha,77Cohorts,6–7,129–130CollaborativeInstitutionalTrainingInitiative,26Commentsinsurvey,152Commercialsurveycompanies,13–14CommonRules,19Comparativeratingscale,53Comparisongroupsurveydesign,7Concurrentvalidity,78,83,130Confidenceinterval(CI),103,147–149Confidentiality,13,19–20Consistency,internal,77–78,83Constructvalidity,78–79,83Contentanalysis,152–153Contentvalidity,78,83Continuousdata,155–156Continuousscales,50Controlgroups,107–109,129Conveniencesampling,94,95,99–102Cookies,23,79Correlations,142–143,154Costello,E.J.,130Costs,survey,11,68CouncilofAmericanSurveyResearchOrganizations(CASRO),29,183Covariate,152Cox,H.,109Credibility,11,29,94,100,181Cross-sectionalsurveydesign,6,7,129Cross-tabs,143–145Culturalbeliefs/expectations

childsurveysand,27,29internationalsurveysand,28–29

Datacategorical,155–156cleaning,163–164

continuous,155–156entering,162–163missing,161–162organizing/managing,157–164protectionof,21–25validatingsurvey,164

DataanalysisANOVA,150,152averages,138–141chi-squaretest,149contentanalysis,152–153correlations,142–143,154cross-tabs,143–145descriptivestatistics,137–138frequencies,137–138,163Mann-WhitneyUtest,150mean,138–139median,139–140mode,140–141nullhypothesis,106,145–146overview,135–136planningfor,7–8poweranalysis,106,108,147range,141regression,142–143risksandodds,150–152selectingmethodsfor,153–157standarddeviation,106–107,141–142statisticalsignificance,146–149summingup,164–165surveyingdifferences,149–152ttest,146,149–150variance,141–142SeealsoData;Results

Datamanagementcodebook/operationalmanualfor,157–159establishingreliablecoding,159–161Kappaand,159–161

Dependentttest,152Dependentvariables,156–157Descriptivedifferences,143–145Descriptivestatistics,137–138Design.SeeSurveydesignDesignweights,110–111Diagrams

drawing,178–179pie,172

Differentialscales,62–63Dirtydata,163Discretescales,49–50Dispersion,106Downey,L.,130Drop-downlist,58

Effectsize,108–109,147Engels,R.,109Environmentalcontrol,surveydesignand,118Equivalencereliability,77ERIC,80Error

marginof,112,113standard,103TypeI,106,147TypeII,106,147

Ethics,survey,19–20childrenand,26–28formalstandardsfor,29onlinesurvey,20–26SeealsoConfidentiality;Culturalbeliefs/expectations;Informedconsent;Privacy

Ethnicity,questionsabout,88Externalvalidity,128–129Eyewitnessreport,5

Face-to-faceinterview,costofconducting,11,68Factorialdesign,124–125FamilyRelationsScale,130Farber,J.,129Figures,listof,insurveyreport,181Film,asinformationsource,5Flesch-Kincaidmethod,60Flowchart,179Focusgroup,11,101Foley,D.L.,130Force-choicequestions,6,52Frequencies,137–138,163Frymethod,60

GallupOrganization,84GeneralSocialSurvey,84Generalusesurvey,18,19Glossaryoftermsinsurveyreport,181Goldston,D.B.,130GoogleScholar,80Graphicscales,50

Graphsbar,172–174,177line,174–178

Hawthorneeffect,128,129,130,131Hewitt,C.E.,109Holloway,I.,132Hypothesis,36–37

Incentives,8,11,12,73,101–102,110,164Independentttest,146,187Independentvariables,156–157Indexinsurveyreport,182Informationneedsselection,36–37Informedconsent,20

forchild,27–28foronlinesurvey,25–26

In-personinterview,telephoneinterviewsvs.,15–18InstituteofEducationSciencesthroughtheNationalCenterforEducation,84InstitutionalReviewBoard(IRB),20,28,29Internalconsistency(IC),77–78,83Internalvalidity,127–128,129,130Internationalsurveys,28–29Internet.SeeOnlinesurveysInterrateragreement,83Interviewers,trainingof,74Interviews,1,5–10

administration,74–75costofconducting,11,68monitoringqualityof,75–76telephone(seeTelephoneinterviews)visualaidsfor,76

Jargon,46Jayakody,S.,109

Kang’ombe,A.R.,109Kappa,159–161Kleinjan,M.,109

Languageandsurveys,28–29Length,survey,68Levelofsignificance,106,146–147Likebest/likeleast(LB/LL)technique,40–44Likert-typescale,52Linegraph,174–178Literaturesearchesandsurveys,79–84

Longitudinal/cohortdesign,6–7,129–130

Mailedinterview,costofconducting,11Mann-WhitneyUtest,150Marginoferror,112,113McNemartest,52,152,154Mean,138–139Measurementvalidity,126,129,130Median,139–140Methodsinsurveyreport,181–182Missingdata,161–162Mobilephonesurvey,14Mode,140–141Multiple-choicequestions,39,44–45

NationalCenterforEducationStatistics,85NationalInstitutesofHealth,26NationalOpinionResearchCenter(NORC),84Nominalresponsescale,49Nonrandomizeddesign,125Nonresponseweights,111Normaldistribution,103,106Normativesurveydesign,125Nullhypothesis,106,145–146Numericalscale,49–50

Observations,5Oddsratio,150–152Onlinelibraries,80,81,82Onlinesurveys

advantagesof,11–12branching,72checklistfor,73commercialsurveycompanies,13–14comparedtomailedsurvey,15–18concernsabout,13–14costs,11,68ethicsand,13,20–26freesurveys,examples,84–85informedconsentand,25–26missingdataand,161–162real-timeinformationand,169–170rulesforcreating,57–60validity,164whentouse,12–13

Open-endedquestions,6,39organizingresponsesto,40–44

writing,38Operationsmanual(codebook),157–159Oralpresentations,184Orderofsurvey,examples,68–71Ordinalscales,49,51–53Otten,R.,109Outcomesinsurveyreport,181

Panels,101–102ParentingPracticesQuestionnaire,130Pearsonproduct-momentcorrelation,142,154Performancetests,5PewCharitableTrusts,84Piediagrams,172Pilottest,8,11,84

guidelinesfor,85–87reliability,86validity,86–87

Piping,72Plus-oneapproach,96Policy/programsneeds,surveysfor,2Posters,188–191Poststratificationweights,111Poweranalysis,106,108,147Predictivevalidity,78,83Presentations,oral,184Privacy,19

childrenand,57onlinesurvey,13,21–25SeealsoEthics

Programevaluationsurvey,2–3,5Progressbar,58Progressgraph,73“ProtectionofHumanSubjects,”19PublicLibraryofScience(PLoS),80PubMed,80Purposivesample,101Pvalue,106,147,148

Qualitativeanalyst,statisticianvs.,152Qualitativesurveydata,validityof,130–132Quality,survey.SeeReliability;ValidityQuasi-experiments,121–122Questionnaires

administrationof,1,72–73format,71overlyambitious,4–5

ParentingPractices,130reproducing,167–169self-administered,4–10,15–18

QuestionPro,169Questions

branching,71–72closed,38,39–40,44–49forced-choice,6,52multiple-choice,39,44–45open-ended,6,38,39,40–44ordering,68–71plainandsimple,60recall,69reliability,39transitions,70writing,44–48

Quotasampling,101

Randomdigitdialing,96Randomizedcontroltrial,124–125,129,130Randomsampling,94–99

defining,7simple,95–96simplerandomcluster,98–99stratified,96–98

Range,141Rankordercorrelation,142Rankorderscale,53Ratingscales,49–53Recommendationsinsurveyreport,182Referencesinsurveyreport,182Regression,142–143Relativerisk,150–152Reliability,76–78

equivalence,77internalconsistency,77–78,83likebest/likeleast(LB/LL)technique,41–42ofpilottest,86ofquestions,39ofsurveyinresearchliterature,82–83test-retest,77validityand,11

Research,surveysfor,3,5Researchliterature,findingsurveysin,79–84Respondents

characteristicsof,82forpilottesting,86

incentivesforparticipating,8,11,12,73,101–102,110,164Response

coding,40–44,159–161formatting,71plainandsimple,60

Responserate,8–9,100,110Results

bargraphs,172–174,177diagrams/pictures,178–179linegraphs,174–178oralpresentationof,184piediagrams,172posters,188–191reporting,9–10reproducingquestionnairesfor,167–169summingup,191tables,169–172usefulness/credibilityof,11writing,179–182SeealsoDataanalysis

Reviews,5Risksandodds,150–152Roadmapinsurveyreport,181Rosen,J.,130Ryan,K.,132

Sample,defining,6.SeealsoSamplingSampling

confidenceleveland,112,113convenience,94,95,99–102finding,102focusgroup,101marginoferrorand,112,113onegroup,nointervention,102–103panel,101–102powerand,106,108purposive,101quota,101random,94–99responserate,tipsforimproving,100simplerandom,95–96simplerandomcluster,98–99sizeofsample,93–95,102–103,106–109snowball,101standarderror,103stratifiedrandom,96–98summingup,113–114

systematic,99,103twogroupsandintervention,103–105weighting,110–111

Scales,rating,49–53Scaling,60–64Schuck,K.,109SecureSocketsLayer(SSL)protocol,21Selecting/adaptingsurveys,79–87Self-administeredquestionnaires,5–10,15–18

administrationof,1,72–73Serviceagreement,79Shelltables,170Simplerandomclustersampling,98–99Simplerandomsampling,95–96Siu,A.,129“Skip”pattern,71–72Slidepresentations,184–186Snowballsampling,101Spearmanrankordercorrelation,142Specificusesurvey,18,19Standarddeviation,106–107,141–142Standarderror,103Stanley,J.,127Statementofpurposes/objectivesinsurveyreport,181Statisticaldifferences,145–146Statisticalsignificance,146–149Statistics,descriptive,137–138Stratifiedrandomsampling,96–98Structuredabstract,180–181Subgroups,identifying,104,105Summaryinsurveyreport,181Summatedscales,63–64Surveydesign

casecontrol,126comparisongroup,7cross-sectional,6,7,129factorial,124–125longitudinal/cohort,6–7,129–130normative,125questionwriting,44–48sampleand,6–7summingup,132–133validityof,126–132

Surveyformavailabilityofinformation,37children,54–57closedquestions,responsesfor,48–54

closedquestions,writing,44–48content/topicfor,35–36informationneeds/hypothesis,36–37onlinesurveys,57–60open-endedquestions,organizingresponsesto,40–44overviewof,35plainandsimplequestions/responses,60questionwriting,38–40,44–48ratingscales,49–53relevanceofinformation,37–38scaling,60–64summaryof,64–65termdefinitions,36

Surveyreport,179–182checklistfor,182–184oralvs.written,187

Surveysadministration,1,72–75cellphone,14childrenand,26–28combinedwithotherinformationsources,5commercialsurveycompanies,13–14comparingtypesof,15–18continuumof,18–19costs,11,68dataanalysis(seeDataanalysis)defining,2–3ethics(seeEthics)format,71generaluse,18,19guidelinesforfindinginresearchliterature,79–84international,28–29language/cultureand,28–29lengthcounts,68online(seeOnlinesurveys)orderof,68–71overviewof,1phone(seeTelephoneinterviews)pilottesting,8,11,84,85–87programevaluation,2–3,5reasonstoconduct,2–3reliability(seeReliability;Validity)reportingresultsof,9–10research,3responserate,8–9selecting/adapting,79–87selectioncriteria,11–18

self-administered,1,5–10,15–18,72–73specificuse,18,19summaryof,30–31tomeetpolicy/programneeds,2translating,87–88validity(seeValidity)whentouse,3–5SeealsoQuestions;Sampling;Surveydesign;Surveyform

Systematicsampling,99,103

Tableofcontentsinsurveyreport,181Tables,169–172

inwrittenandoralreports,187,190listof,insurveyreport,181preparationrules,171–172

Telephoneinterviewsadministrationof,74cellphones,14costs,11,68in-personinterviewvs.,15–18orderingquestionsfor,69samplingfor,96

Test-retestreliability,77Tilbrook,H.E.,109Training,interviewer,74Transitions,70Translatingsurveys,87–88Trueexperiments,122–124Ttest,145,146,149–150,152,153,154,187Two-by-threefactorialdesign,124Two-by-twofactorialdesign,124TypeIerror,106,147TypeIIerror,106,147

Unimodaldistribution,140U.S.Census,84

Validityconcurrent,78,83,130construct,78–79,83content,78,83external,128–129internal,127–128,129,130measurement,126,129,130ofonlinesurvey,164ofpilottest,86–87ofqualitativesurveydata,130–132

ofsurveydesign,126–132ofsurveyinresearchliterature,82–83predictive,78,83reliabilityand,11surveydata,130–132,164threatsto,127–131

VanTeijlingen,E.,132Variables,156–157Variance,141,142,145Variation,141–142Videotape,asinformationsource,5Visualaids,76

WebofScience,80,81Websurveys.SeeOnlinesurveysWeighting,110–111Wilcoxonranksum.SeeMann-WhitneyUtestWriting

oralvs.writtenreports,187questions,44–48surveyresults,179–182tips,183–184

Writtentests,5

Zun,L.,130

Date post:	19-Mar-2018
Category:	Documents
Upload:	duongkhue
View:	220 times
Download:	5 times

How To Conduct Surveys - Home | 2017 - UoC Exams | · PDF file · 2017-01-28How To...

Documents