Survey Design & Item Construction

Lindsay Couzens, M.S.UNLV’s 2009 Academic Assessment

WorkshopMay 14th & 15th

Developing a Purpose

What is your overall research question? What questions do you want your survey to

answer? Is there an existing survey that answers

your questions? Keep length of the survey in mind

Define target group

Identify Objectives2UNLV Office of Academic Assessment

Question Objectives

Identify Objectives Based on purpose/research question Example: Fowler, 1995

Develop definitions for objectives All respondents should have a common

understanding of the meaning of each question

Specify the kinds of answers needed to meet the objectives Informs item construction

3UNLV Office of Academic Assessment

Types of Surveys

In person Can clarify confusing

statements Immediate feedback Limited to those who are

present Mail

Can reach large geographic areas

Depend on respondents’ motivation

Requires current address database

Email Can reach large geographic

areas instantly

Cost-efficient Requires computer access

Online Cost-efficient Requires computer access

Interview Can ask exploratory or

probing questions Can assist with unclear

questions Requires trained

interviewers Requires up-to-date

telephone numbers

4UNLV Office of Academic Assessment

Issues to Consider

Timeline Number of data collection points Cutoff dates

Cost

Who will construct the survey? How? ▪ Survey Monkey▪ Elisten▪ WebCampus tool

5UNLV Office of Academic Assessment

Issues to Consider

How will the survey be distributed?

Who will analyze the survey? Do you need special hardware or

software?

Who will maintain the survey? Who is responsible for updating,

revising, editing?

UNLV Office of Academic Assessment 6

Methods

Pilot testing

Focus groups

How will the surveys be distributed?

How will they be collected?

7UNLV Office of Academic Assessment

Research Questions and ObjectivesThomas, 1999

What is the research question?

Do the objectives for the survey flow from the research question?

Is each objective stated in concrete terms?

Does each objective measure one idea or concept?UNLV Office of Academic Assessment 8

Item Construction

Question Content

Each question should be mapped to one of your identified and defined objectives Consider the content, scope and purpose of

each item

Response formats Open-ended or closed-ended?

Survey layout Items should be grouped by topic AND by

response type10UNLV Office of Academic Assessment

Response Formats

Things to Consider Clarity of Directions Levels of measurement▪ Nominal▪ Ordinal▪ Interval

Exhaustiveness▪ Response alternatives have sufficient range

11UNLV Office of Academic Assessment

Response Formats

Things to Consider Exclusiveness▪ Response alternatives are mutually

exclusive

Balancing Categories▪ There should be an even number of

alternatives on both sides of neutral

UNLV Office of Academic Assessment 12

Response Formats

Numerical Rating Scales Likert-type scales▪ Rates attitudes or opinions

Horizontal rating scales▪ Used to indicate where a respondent’s

attitude or opinion falls between two opposite attitude positions

13UNLV Office of Academic Assessment

Response Formats

Numerical Rating Scales Semantic differential▪ Opposite adjectives rather than

attitude positions Vertical rating ladder▪ Ranking

UNLV Office of Academic Assessment 14

Response Formats

Scores Out of 10▪ Respondents rate each item

Ranking Items are ranked relative to other items

in the set

15UNLV Office of Academic Assessment

Response Formats

Checklists Respondent checks all that apply

Binary choice formats Dichotomous questions▪ Ex: What is your sex? (M/F); Do you smoke?

(Y/N) Paired comparisons▪ Respondent chooses between two

overlapping optionsUNLV Office of Academic Assessment 16

Response Formats

Multiple choice formats Choice between multiple nominal

categories▪ Marital status

Choice between ordinal categories▪ How often do you eat in the dining

commons?

17UNLV Office of Academic Assessment

Response Formats

Multiple choice formats Choice between ordered attitude

statements▪ Which attitude statement most closely

matches your own?

Numerical answers▪ Salary; how many people live in the

householdUNLV Office of Academic Assessment 18

Question Wording ChecklistdeVaus, 2002

Is the language simple?

Can the question be shortened?

Is the question double-barreled?

Is the question leading?

Is the question negative?19UNLV Office of Academic Assessment

Question Wording ChecklistdeVaus, 2002

Is the respondent likely to have the necessary knowledge?

Will the words have the same meaning for everyone?

Is there a prestige bias?

Is the question ambiguous?20UNLV Office of Academic Assessment

Question Wording ChecklistdeVaus, 2002

Is the question too precise?

Is the frame of reference for the question sufficiently clear?

Does the question artificially create opinions?

Is personal or impersonal wording preferable? 21UNLV Office of Academic Assessment

Question Wording ChecklistdeVaus, 2002

Is the question wording unnecessarily detailed or objectionable?

Does the question have dangling alternatives?

Does the question contain gratuitous qualifiers?

Is the question a “dead giveaway”?22UNLV Office of Academic Assessment