DEVELOPING A QUESTIONNAIRE FOR

USE IN OUTCOMES ASSESSMENT

Kathleen A. Martin, D.P.E.Office of Institutional Research & Assessment

Bucknell University

POINT OF CLARIFICATION

Terms “survey” and “questionnaire” are often used interchangeably

“Survey” refers to a descriptive research technique or methodology (also includes interviews, and focus groups)“Questionnaire” refers to a paper and pencil instrument through which information is obtained

For the purposes of outcomes assessment, questionnaires are frequently developed to survey a population

QUESTIONNAIRES & ASSESSMENT

Useful for surveying groups of any sizeTypically designed to determine opinions, attitudes or present practicesLimitation: rely on self report information v. observed behavior

Challenges to validity become a concern

VALIDITY CHALLENGESValidity: the extent to which the test measures what it purports to measureInternal Validity: Relates to instrument (questionnaire) quality. Are the appropriate questions asked clearly and logically?External validity: can the results be generalized?

Measurement error: difference between the characteristics of your sample and the characteristics of the population

Non-response bias: answers of respondents do not mirror the characteristics of those who refused participation.

MAXIMIZING VALIDITYCreate a questionnaire that is clear and logical

Support content relevance and content representativenessContent Relevance: are the questions relevant to the purpose of the questionnaire? Unimportant content should be eliminated.Content Representativeness: Are the questions an adequate representation of the universe of possible questions?Solicit the input of experts in scale development and the content of the questionnaire for an objective evaluation

Select a large, representative random sample from the population of interest or census the finite populationIncrease response rate using formal follow up procedures

INITIAL CONSIDERATIONS

Is the development of a new questionnaire . . .

Necessary?Feasible?

Select mode of administrationWeb Based?Paper and pencil?Combination?

QUESTIONNAIRE PLANNING

1. Determine the ObjectiveWhat do you want to ask and of whom?Consider how the data will be analyzed. What information is needed?

2. Determine the Mode of AdministrationPaper and pencil

Cost effective, but low response rateWeb / E-mail based

Cost effective, higher response rate, some suspicion of the medium

CombinationCan be administratively cumbersome

QUESTIONNAIRE PLANNING

Determine the Sampling MethodologyTo whom will you send the questionnaire? Who can supply the information?Will everyone be surveyed?

ALL current students? ALL alumni?If yes, this is a census of the finite population

The selection of a representative sample of the population may be more effective based on time and cost of administration

Decision depends on the size of the finite population

CONSTRUCTING the QUESTIONNAIRE

Consider what objective each question measuresThree to five questions per objective is recommended

How many questions?Find a balance: long enough to obtain the necessary information but short enough so respondents won’t lose interest.Questionnaires that are too long won’t be completedGoal: 25-30 minutes MAX

DESIGNING THE QUESTIONNAIRE

Question Formats: Open Ended QuestionsAllow respondents to express feelings and expand on ideas. Question should be phrased to avoid a one word response.

“What aspects of the Sociology program best prepared you for your current position?”

Open ended questions at the beginning of the questionnaire may be discouraging to respondent – include one or two at the endAnalysis of responses can be challenging

QUESTIONNAIRE DEVELOPMENTQuestion Formats: Closed Questions

RankingForces the respondent to rank order responses based on a value judgment. Rank the following items with regard to how you spend time at Bucknell from 1 (spend the most time) to 5 (spend the least time).

_____ Studying_____ Socializing_____ Volunteering_____ Working at a part time job_____ Exercising

QUESTIONNAIRE DEVELOPMENTQuestion Formats: Closed Questions

ChecklistA number of possible answers are provided and the respondent is asked to choose one or choose all that apply. How did you finance your Bucknell University education? Choose all that apply.

_____ Parental contribution_____ Student Loan_____ Work Study_____ Off campus employment_____ Other (please specify): _____________

QUESTIONNAIRE DEVELOPMENTQuestion Formats: Closed Questions

Scaled itemsVery common

Indicate strength of agreement or disagreement with a statement.

Numbers are assigned to each response on the continuum

The English curriculum at Bucknell University adequately prepared me for my current position. 1 2 3 4 5

Strongly

Disagree Disagree

Neutral / No

Opinion AgreeStrongly

Agree

QUESTIONNAIRE DEVELOPMENTQuestion Formats: Closed Questions

Suggestions for writing scaled itemsUse positively worded or negatively worded items consistently

Use an odd number of responses along the continuum to allow for a neutral response

Anchors may be different based on focus of the item (Strongly Agree – Strongly Disagree; Frequently – Never; Very Good – Very Poor)

Keep items short

QUESTIONNAIRE DEVELOPMENT

Question Formats: CategoricalTypical demographic questions:

Yes/No/?, Male/Female, F/S/J/S/GAnswers fall into certain categories – respondents pick the one that applies to themAnalyzed using frequency, proportion, percentages2 considerations:

Questions on a continuum should stay on a continuum if they will be directly analyzed

Example: Age – allow respondent to write in their age rather than check off a box for their age range

Categories must be collectively exhaustive & mutually exclusive (next slide)

QUESTIONNAIRE DEVELOPMENT

Problematic:Number of hours spent studying per week:

18 – 21 21 – 2424 – 27

Categories are not mutually exclusive

Corrected:Number of hours spent studying per week:

18 – 2122 – 2526 – 29

IMPORTANT CONSIDERATIONS

APPEARANCE AND DESIGNHave clear directions that are age appropriate & easy to followFormat, size, reproduction should facilitate completion

No less than 12 ptTimes, Arial, Courier

PILOT STUDYGive the questionnaire to a few people to try it outHave them evaluate readability and clarity of questionsEstimate time required to complete the questionnaireSuggestions for revisions

SENDING IT OUTINITIAL CONTACT / COVER LETTER

Short & to the pointIdentify person by nameConvince respondent that participation is importantEnsure confidentialityDeadline for return of questionnaireSASE for return (mailed questionnaires only)

Institutional approvalInstitutional Review Board

Research involving human subjects must be approved by the Bucknell University IRBFor more information, visit the IRB website at http://www.departments.bucknell.edu/inst_research/IRB/irb.htm

Campus Survey CoordinationThe timing of all university-wide surveys must be coordinated through the Campus Survey Coordination Group.For more information, visit the CSCG website at http://www.departments.bucknell.edu/inst_research/cscg.htm

FINAL THOUGHTSFOLLOW UP PROCEDURES

Send to all participants to preserve confidentiality (“If you haven’t already returned the questionnaire…..”)May include a second copy if availableUsually done in 2 week intervals

RESPONSE RATE FOR MAILED QUESTIONNAIRES

Typically low - 30% is considered goodHigher response rate with web-based questionnairesGood follow-up increases response rate

SUMMARY OF STEPS1. Determine the Objective (consider analyses)2. Determine the Mode of Administration3. Determine the Sampling Methodology4. Construct the Questionnaire5. Institutional Approval6. Conduct the Pilot Study7. Write the Initial Communication 8. Send the Questionnaire9. Follow up10. Analyze the Results

FOR MORE INFORMATION

Contact Kathy Martin @ 71960 or kmartin@bucknell.edu