Needs Assessment:Needs Assessment:Surveys Surveys

Dale A. Moore, DVM, PhD

School of Vet Med, UC Davis

ObjectivesObjectives

Describe the needs for needs assessments

Describe one way to conduct a needs assessment

Provide some examples

Needs For What?Needs For What? Program topics Location Delivery Method Etc…

A need is a discrepancy between what the desired goal or standard is with what actually exists.

Program DevelopmentProgram Development Identification of the Main Themes*** Description of the Program Purpose Conduct Needs AssessmentConduct Needs Assessment*** ID others interested in development &

delivery of the program*** Outline learning objectives*** Select format and teaching methodologies*** Select faculty / facilities*** Evaluation tool*** Marketing and promotion strategy*** / budget

Nancy Bennett, Dev CPE Programs, 1990

Traditional CETraditional CE Lecture-dominated format Little evidence of impact on behavior or

outcomes Minimal collaboration between learners and

providers Lack of timely response to learner needs Emphasis on credit Focus on course production driven by

enrollment economy

Don Moore, An approach to needs assessment in Physician practices. Alliance for CME, 1998

New Paradigm in CENew Paradigm in CE Increased emphasis on learningData-based educational activitiesCollaborative learning systemsBlending of quality mgmt and CMEFocus on improving patient outcomesCME as integral part of healthcare

systemDon Moore, An approach to needs assessment in

Physician practices. Alliance for CME, 1998

The Need For Needs The Need For Needs AssessmentAssessment ID learning objectivesData-driven education ID the standards of practice/knowledge ID the goals for outcomes &

stakeholder expectations ID the role of the educational program

in the practice of XXXXX

Needs Assessment for Needs Assessment for Better Evaluations!Better Evaluations!

1

11 1

Needs Assessment: Which Way to Go?Needs Assessment: Which Way to Go?

Different Levels of NeedsDifferent Levels of Needs Societal needs / client /

industry needs and expectations

Community needs Profession’s needs /

professional sector’s needs Farm’s needs Individual’s needs

Needs Assessment MethodsNeeds Assessment Methods Environmental Scans

Review of literature Discussion with experts

SurveySurvey Mail surveyMail survey Telephone surveyTelephone survey Face-to-face interviewsFace-to-face interviews

Focus Groups Clinical audits / Practice audits Self-assessments

SurveysSurveysAdvantage of providing quantifiable

responses from randomly-selected study participants

Usually require that the response choices are known by the investigator in advance

Issues of response-rates

List Frame for Surveys: WHO?List Frame for Surveys: WHO?

Your own/other database of previous participants

Company mailing listsOthers?

Selection From the ListframeSelection From the Listframe How to randomly select?

Random number generator in Excel Tools/Data Analysis/Random Number

Generator/OK/Variables…… Sample size? EpiCalc 2000

http://www.brixtonhealth.com/epicalc.html

Designing Your Survey and Designing Your Survey and Survey QuestionsSurvey Questions

Fowler, FJ. (1995). Improving Survey Questions: Design and Evaluation. Thousand Oaks: Sage Publications.

Dillman, DA. (2000). Mail and Internet Surveys: The Tailored Design Method. John Wiley and Sons: New York.

Online SurveysOnline SurveysANR Site Builderhttp://groups.ucanr.org/help/index.cfmThe mechanicsmechanics of putting a survey

together on-linehttp://www.surveymonkey.com/

Improving Response RatesImproving Response RatesFollow Dillman’s MethodKeep your promises to your

constituents Incentives – CASHMultiple contactsRespondent-friendly questionnaires

Improving Response RatesImproving Response RatesThe number of contacts Advance

letters, postcards, follow-up, additional copies of questionnaires, telephone calls (5)

Salience of the topic Relevance. Common reason for non-response is that survey doesn't mean anything to the person.

Improving Response RatesImproving Response RatesGovernment sponsorship: higher

response rates than private organizations.

Special third contact Follow-up of advance letter / initial follow-up with special mailing procedures: certified mail or special delivery, personal or telephone contact.

Incentive on first contact: Incentives included with first mailing increase response rate. Incentives offered later not effective. ($5 cash)

What does good questionnaireWhat does good questionnairedesign involve?design involve? • IDENTIFY MEASUREMENT OBJECTIVES! • Draft questions based on objectives • Test questions – Expert review – Cognitive interviewing – Pilot run • Format of questionnaire

Respondent-Friendly Q’airesRespondent-Friendly Q’aires

Not too longAsk most important questions first;

demographic information lastUnambiguous questions/responsesEasy to completeGreat directions -- subheadings

Critiquing/Drafting QuestionsConsistently understood (avoid

ambiguous words, define key terms)Avoid “double-barreling” "(asking two

questions at once) “Would you like to be rich and famous?”

Avoid hidden assumptions Consistently administered and

communicated – interviewers read all response options

Some Help With Questionshttp://www.statpac.com/surveys/

surveys.dochttp://www.leeds.ac.uk/iss/

documentation/top/top2.pdfhttp://home.ubalt.edu/ntsbarsh/stat-

data/Surveys.htm#rsiMany others and books, like Dillman

9 rules for writing good 9 rules for writing good questionsquestions

1. Remember your survey's purpose

2. If in doubt, throw it out3. Keep your questions simple

Example:Example: "Imagine a situation where the

production supervisor is away from the line, a series of defective parts is being manufactured, and you just heard that a new client requires ten thousand of these parts in order to make their production schedule. How empowered do you feel by your organization to stop the line and make the repairs to the manufacturing equipment?"

4. Stay focused - avoid vague issues"When did you last see a movie?"

5. If a question can be misinterpreted, it will be"What time do you normally eat dinner?"

6. Include only one topic per question

7. Avoid leading questions "Most doctors believe that exercise is good for you. Do you agree?"

8. Consider alternate ways to ask sensitive questions Income, drug or alcohol consumption and sexual habits.

9. Make sure the respondent has enough information

5 rules for obtaining usable answers 1. Response options mutually exclusive and

exhaustive. 2. Keep open-ended questions to a

minimum they pose problems in terms of coding and analysis.

3. People interpret things differently, particularly when it comes to time Trouble-spots include responses such as "Always," "Sometimes" and "Never." Build in a temporal frame of reference. "I am going to read a list of publications. For each one, please tell me whether you read it regularly. By regularly I mean, at least three out of every four issues."

4. Consider a "Don't Know" response Allow people to say they do not have an opinion.

5. Provide a meaningful scale End points must be anchored with labels. “Rate your satisfaction where 1 means 'Very Satisfied' and 5 means 'Very Dissatisfied.‘ Odd number of points provides middle alternative. Gain nothing by having scale with > 7 points.

Survey LengthSurvey Length30-40 questionsFewer the betterBooklet form?

Survey FlowSurvey Flow

Pilot-TestingPilot-TestingWith peersWith a small group of your audienceHelps refine the tool

Surveys for Needs AssessmentSurveys for Needs Assessment

What are your outcomes?What is your audience?Sampling frame, sample size, how to

select randomlySurvey distributionQuestionnaire designData analysis