As the most commonly used indicator is a survey/questionnaire, the following power point provides detailed instructions for construction of valid indicators using this data collection technique. Why is it important to construct valid indicators? Garbage in – Garbage OUT! Your indicator may appear professional, but if it is not valid, you have garbage. You may have a large sample size, but if your indicator is not valid, you have garbage. You may use “high powered” statistics, but if your indicator is not valid, you have garbage. You may have a great program, but if your indicators are not valid, then your evaluation of the program is garbage. You may have a well-written journal article, but if your indicator is not valid, you have garbage.
Transcript
As the most commonly used indicator is a survey/questionnaire, the following power point provides detailed instructions for construction of valid indicators using this data collection technique.
Why is it important to construct valid indicators? Garbage in – Garbage OUT!
Your indicator may appear professional, but if it is not valid, you have garbage.
You may have a large sample size, but if your indicator is not valid, you have garbage.
You may use “high powered” statistics, but if your indicator is not valid, you have garbage. You may have a great program, but if your
indicators are not valid, then your evaluation of the program is garbage.
You may have a well-written journal article, but if your indicator is not valid, you have garbage.
STEPS TO CONSTRUCTING A STEPS TO CONSTRUCTING A SURVEY INSTRUMENTSURVEY INSTRUMENT
Step 1 – Make Decisions about TYPES of Indicators Level of measurement Single vs. multiple indicators Closed vs. open-ended indicators
Step 2 – Consider Content of Questions Make it easy for respondents Make it easy for those who input data Construct indicators that yield reliable
(consistent) and valid (accurate) results
Step 3 – Consider Format of Survey/Questionnaire
Step 4 – Know How to Evaluate Surveys
Step 5 – Construct a Codebook
STEP 1 – MAKE DECISIONS STEP 1 – MAKE DECISIONS ABOUT TYPES OF INDICATORSABOUT TYPES OF INDICATORS
• Level of measurement• Measure at highest level possible.
Determined by1. How precise cancan respondents
be?• For example - Can Can they
remember their exact income?
2. How precise are they willingwilling to be? • For example - Will Will they tell
you how many STIs they have?
• Single vs. multiple indicators• Multiple indicators should be used
to measure different dimensions of same concept or ask about multidimensional concept.
• Examples1. Scales used to measure marital
happiness, depression, college readiness (i.e., SAT, ACT), etc.
2. Questions used to measure different dimensions of social class (e.g., income, occupation, education)
STEP 1 – MAKE DECISIONS STEP 1 – MAKE DECISIONS ABOUT TYPES OF INDICATORSABOUT TYPES OF INDICATORS
• Close vs. open-ended questions• Two types of closed ended questions
1. Unordered - Ask about typetype of activity
Example - Which of the following activities do you engage in before going to bed?
1. Reading2. Eating3. Both of the above
2. Ordered - Ask about degree or strength of reaction degree or strength of reaction to statement
Example - Indicate how satisfied you are with the following:
VD DS S VSUSU Extension Services 1 2 3 4
County Extension Agent 1 2 3 4
STEP 1 – MAKE DECISIONS STEP 1 – MAKE DECISIONS ABOUT TYPES OF INDICATORSABOUT TYPES OF INDICATORS
• Close vs. open-ended• Open-ended questions
• What are the advantages?• Important for exploratory
research . For instance, use when you do not yet know WHAT questions to ask.
• Provide more in-depth information about topic.
• What are the disadvantages?• Time consuming. Coding is
required once data is collected• Validity and reliability are
dependent on the coders and their skills.
• Examples of open-ended questions commonly used in evaluation research• What are the strengths of this
program?• How would you improve this program?• What things have you learned from
this presentation that you can share with others?
STEP 2 – CONSIDER CONTENT STEP 2 – CONSIDER CONTENT OF QUESTIONS OF QUESTIONS
Make it easy for respondents Ask questions at the 6th grade level – 20 words
or less
Avoid the word NOT in negatively worded questions (i.e.,
I do NOT study every night.)
Avoid Acronyms (i.e., USU, ARD)
Avoid putting blanks in the middle (i.e., I read for pleasure_____ times per week.)
Avoid having respondent rank items (i.e., rank ten most serious crimes from high to low.)
When using contingency questions mark them clearly with arrows
Match questions/statements with respondents in terms of
1. Interests – Don’t ask questions about something in which they have NO interest.
2. Competency - Don’t ask questions about things they know nothing about.
3. Relevance - Don’t ask questions about things that aren’t relevant to them and/or their situation.
STEP 2 – EXAMPLES OF SUGGESTIONS ON STEP 2 – EXAMPLES OF SUGGESTIONS ON SLIDE 5 SLIDE 5
Making it easy for respondent
Write at 6th grade reading level – 20 words or lessX Instead of
X Based on your annual net income for the previous decade, excluding any atypical pay periods or wages accrued by minors living in your household, what would be your best approximate estimate of your socio-economic status?
X A better alternative might beX What is your social class?
Avoid the word NOT in negatively worded questions
I very seldom can’t stop worrying about my grades.
A better alternative would be “ I often worry about my grades.”
This professor is not a competent teacher. A better alternative would be
This professor is an incompetent teacher
Avoid Acronyms Do you think CEOs who graduated from USU
should attend ARDs if their child has ADD or an STI?
Avoid putting blanks in the middle I spend ____hours studying for exams.
A better alternative would be On average, how many hours do you
spend studying for exams during a typical week? ____________ hours
STEP 2 – EXAMPLES OF SUGGESTIONS ON STEP 2 – EXAMPLES OF SUGGESTIONS ON SLIDE 5 SLIDE 5
Making it easy for respondent
Avoid having respondent rank items Rank the following leisure activities from 1
to 8 based on how much you enjoy them.__Read books __E-mail a
friend__Ride a bike __Eat potato
chips__Swim __Talk
on my cell phone__Watch T.V. __ Gossip
X A better alternative might be Please indicate your two most favorite
leisure time activities (1) ________, (2)________
Use contingency questions marked clearly with arrows.
Match questions/statements with respondentsX Interests – Don’t ask teenagers about flat
taxes or retirement pensions.X Competency – Don’t ask the yearly income
of their great Uncle Fred.X Relevance – Don’t ask singles about their
Make it easy for those who code, input and /or interpret data. You cannot input letters or boxes into the computer. The computer only recognizes numbers. THEREFORE, have respondents circle numbers
for their answers. Avoid letters and boxes.
For answer categories make certain HIGH numbers correspond to POSITIVE answers. When doing analysis, it is much easier to
interpret a number if a high number corresponds to a positive answer. That is the way we are trained to think about numbers.
Use open-ended questions only when necessary. When possible use a series of open-ended
questions to get more precise answers.
STEP 2 – EXAMPLES OF SUGGESTIONS ON SLIDE 8 STEP 2 – EXAMPLES OF SUGGESTIONS ON SLIDE 8
Making it easy for you Have respondent circle numbers for answer Have respondent circle numbers for answer
categoriescategories. Do you grow corn? ____ Yes _____NoA better alternative would be
Do you grow corn? (1) Yes (2) No
Have high numbers = positive answersHave high numbers = positive answersWhat grade did you receive in this class last semester?
(1) A (3) C (5) F(2) B (4) D
A better alternative would be What grade did you received in this class last semester?
(1) F (3) C(5) A (2) D (4) B
Please indicate how satisfied you are with the information you received from this presenter. Possible answers range from (1) very satisfied to (4) very dissatisfied.
VS S DS VD 1 2 3 4A better alternative would bePlease indicate how satisfied you are with the information you received from this presenter. Possible answers range from (1) very dissatisfied to (4) very satisfied.
VD DS S VS1 2 3 4
STEP 2 – EXAMPLES OF SUGGESTIONS ON STEP 2 – EXAMPLES OF SUGGESTIONS ON SLIDE 8 SLIDE 8
Making it easy for you
When possible, use a series of open- ended questions to get more precise answers. This can make it much easier to code and interpret your findings.
What do you do for a living? ________
A better alternative might be to ask a series of questions
Where do you work? ____________________
If you work for a company or organization, what is the name of that company/organization? _____________
In a typical day, what types of tasks do you complete?____________________________
Increase Validity/Reliability Wording of Question
Be as Precise/specific as possible When asking about frequency, ask
for a specified time (i.e., On average, how often,
per week do you telephone your mother? ___)
When asking for a number, ask for as precise a number as possible
(i.e., How much time do you spend watching TV during an average day? ____ hours ____ minutes)
When asking for a number, try to be exact
(i.e., How old were you on your last birthday?)
Increasing Validity/Reliability Wording of Question
Avoid Slang – “hot” “cool” Do you think it is cool to skip class? A better alternative might be
Do you think students approve of skipping classes?
Avoid Bias or Leading Questions• Cancer is a devastating illness that kills
millions of Americans every year. How much money do you think we should spend finding a cure for this illness?
• A better alternative might be How much money do you contribute to
cancer research? ________ dollars per year.
Women who are raped never recover. They only survive. What penalty do you think rapists should receive?
A better alternative might be Indicate the extent to which you agree with
the following statements: Current penalties for convicted rapists are
just. Penalties for convicted rapists should be
more severe. Penalties for convicted rapists are too harsh.
Increasing Validity/Reliability Wording of Question
For Indices Use statements – not questions
SD D NAND A SA
Do you enjoy the firework 1 2 3 4 5
at the county fair?
A better alternative would be
SD D NAND A SA
I enjoy the fireworks 1 2 3 4 5
at the county fair.
Increasing Validity/Reliability Wording of Question
For Indices Avoid response set
SD D ASA
I loved this program. 1 2 3 4
The teachers were veryfriendly 1 2 3
4I learned a great deal. 1 2 3 4I want to do this again 1 2 3 4
A better alternative would beSD D A
SAI loved this program 1 2 3
4The teachers were unfriendly 1 2 3 4I learned nothing new 1 2 3 4I want to do this again 1 2 3 4
Increasing Validity/Reliability Wording of Question
For Indices Avoid using NOT in a negative
statement SD D A
SA
I did NOT like the
program 1 2 3 4
The teachers were Not
friendly 1 2 34
I did NOT learn anything 1 2 34
A better alternative would be
SD D ASA
I disliked this program 1 2 3 4
The teachers were
unfriendly 1 2 3 4
I learned nothing new 1 2 3 4
Increasing Validity/Reliability Wording of Questions
For indicesUse multiple statements to measure
multidimensional concepts (i.e., social distance)
SD D NADA A SA
I would live with an Asian 1 2 3 4 5
I would go to parties with Asians 1 2 3 4 5
I would feel uncomfortable sharing an office with an African American 1 2 3
4 5
My family would disapprove if 1 2 3 4 5
If I married a Hispanic
Increasing Validity/Reliability Wording of Questions
For Indices Avoid double barreled/double blind
statementsSD D A
SAMen and women are
poor communicators 1 2 3 4I like the staff and the 1 2 3 4 teachersMy health has changed 1 2 3 4 since last year
A better alternative might be SD D A
SA Men are poor communicators 1 2 3 4 Women are poor communicators 1 2 3
4 I liked the staff . 1 2 3 4 I liked the teachers 1 2 3 4 My health is better 1 2 3 4 My health is worse 1 2 3 4
Wording of Answer Categories Exhaustive
Cover all possible answer
Word questions so all answers are possible.ExampleHow often, IF EVER, do you attend football games? ____
times per semester.
For instanceUse the answer category “Other (specify)______” whenever it is appropriate.
Word answer categories so all possible answers are acceptable.
For instance,Hitler was a
1. Very negative role model 2. Negative role model
A better alternative would beHitler was a1. Very negative role model 3. Positive role
model2. Negative role model 4. Very positive role
model
Wording of Answer Categories
Use ALWAYS and NEVER with great caution
Please indicate the extent to which you do the following. Possible responses
are (1) Never, (2) Seldom, (3) Sometimes, (4) Often, and (5) Always
N S ST O AI read the newsletter mailed 1 2 3 4 5out by the extension agent.
A better alternative would be to replace NEVER and ALWAYS with ALMOST NEVER and ALMOST ALWAYS.
AN S ST O AA I read the newsletter mailed 1 2 3 4 5
out by the extension agent.
Wording of Answer Categories Mutually exclusive
NEVER have two possible right answers For instance - How much money did you make last
year? (1) $10 ,000 to $20,000 (2) $20,000 to $40,000 (3) $40,000 to $60,000
A better alternative would be (1) $10,000 to
$19,999 (2) $20,000 to
$39,999(3) $40,000 to
$59,999
Wording of Answer Categories
Avoid offensive/biased answer categories
For instance Use overweight NOT obese Use pro-life NOT anti-
abortion Use pro-choice NOT pro-
abortion Use thin NOT skinny Use government assistance NOT welfare
For instance What do you do when you don’t go to
class?(Circle all that apply)
1. Act Like a Lazy Slob2. Waste my time
A better alternative would beHow do you spent your time when you don’tattend class? ________________________
Wording of Answer Categories Be as precise/specific as possible
Using YES/NO as answer categories Disadvantages Doesn’t give you
much variation. Is not precise enough.
Advantages Sometimes it is the
only plausible answer.
Using “Don’t Know” as an answer category
Disadvantages Sometimes, it is NOT a plausible
answer. Can result in response set, as respondents may see it as the
easier, though not most accurate
answer. Advantages
Can add important variation to possible answers, and REMEMBER you
need/ want variation when you start
doing analysis.
General Guidelines Using Numbers
For single items – number from top to bottom
in columns. What grade did you receive?
1. F 4. B2. D 5. A
3. C
For index : (1) put numbers under symbols, (2) number from left to right (3) number from lower to higher
SD D NAD ASA
1 2 3 4 51 2 3 4 5
Professional Appearance Spacing – even and balanced Uncluttered appearance No spelling or grammar errors Can use shading, etc.
Required Sections Textbox
Should contain the following information General statement about the purpose of
survey Statement indicating the time survey will
take Statement that indicates it is anonymous
or confidential Statement that indicates it is voluntary Contact information for those wanting
more information
Basic socio-demographic questions
Questions measuring indicators/variables
Contingency questions All parts should be on the same page
Indices/Scales The entire index/scales should be on same
page – if possible. If you put it on two pages, then repeat headings.
Section Placement
Textbox Top of first page of survey
Sociodemographic First page of survey
Questions measuring indicatorsSensitive questions Bottom of last page of surveyIndependent variable First page of surveyDependent variable Second page of survey
Open-ended questions End of survey
Contingency Questions Throughout the survey
Index (ices) Generally middle or endof survey
Appearance Does it look professional? Is the text “balanced”? Did you use numbers? Are they in correct order?
Statement at Top Does it include relevant/complete information? Is it in a textbox?
Questions Format – for index or scale
Does it have a textbox at top with instructions? Is it numbered from left to right – lower to higher? Did you use statements – not questions? Are statements logically related to each other?
Wording Did you use a spelling and grammar check? Did you avoid – biased statements/word, slang, abbreviations, double-barreled
questions/statements, etc.? Answer categories
-measured at highest level possible?
-exhaustive/mutually exclusive?
Do you test your hypotheses/measure your indicators?
Use numbers not letters
Measure indicators/variables at the highest level possible (i.e., the ratio level, whenever possible)
Ask only relevant questions Don’t ask college students if they graduated from high
school
Use statements rather than questions in indices. You can’t agree with a question!
Avoid bias. If you can predict what answer the researcher “wants” then it is a biased statement or question.
Make wording across answer categories consistent For instance, If you use pro-life, then use pro-choice For instance, If you use African American, then use
Asian American.
You can use the following chart or one like it to evaluate the validity of the indicators used in your survey/questionnaire
Concept Instrument Face Validity (Y/N)
For each indicator, ask yourself the following question?1. Can you match a question or questions on your
surveywith each indicator you are using to measure impact?
2. Are you measuring attitudes or behavior? Does the question/statement in the survey match?
2. If you are measuring a multi-dimensional concept – doyou have more than one question/statement?
Then determine if your concept has face validity (i.e., it logically appears to measure the concept).
You can use the following chart or one like it to evaluate the validity of the indicators used in your survey/questionnaire
Concept _________ Instrument _________ Face Validity (Y/N)
Impact of program How often do you recycleY
on recycling behavior your drink containers?
Impact on attitudes toward To what extent do you agreeY
water conservation that Utah residents must learn to conserve water or
face shortages?
For each indicator, ask yourself the following question?1. Can you match a question or questions on your survey
with each indicator you are using to measure impact?
2. Are you measuring attitudes or behavior? Does the question/statement in the survey match?
2. If you are measuring a multi-dimensional concept – doyou have more than one question/statement?
Then determine if your concept has face validity (i.e., it logically appears to measure the concept).
You can use the following chart to evaluate the reliability of each of your indicators. Take each concept, and then rate the indicator on the following characteristics. Generally, the higher the score, the more reliable the indicator.
For each of your indicators, ask yourself if you think your indicators would yield consistent results across time. Answer the following questions.
1. Are questions precise and unambiguous?2. Are there any words that may be interpreted
differently by different people? (slang)3. Are there biased questions that may influence some
people’s answers?4. Are any questions double barreled?5. Are answers exhaustive? (Did you use “Other –
please specify” as an answer category?)6. Are answers mutually exclusive?7. Are there questions that may offend some?
General Guidelines
Identify a name for your codebook. It should start with a letter (not a number), and should contain no slashes, hyphens, etc. It should be about 8 characters long.
Put an identification number on each of your surveys. This identification number should be the first variable in your codebook.
Construct a name to identify each statement/question used in your survey to measure each indicator. Do not use slashes/dashes/spaces. This name will appear on your statistical outputs when analyzing this question/statement.
Compose a BRIEF description for each variable/indicator.
Construct a code that corresponds to each variable/indicator. The code should consist of numbers representing each possible response/answer.
Q # Var# Var Name Description ___ Code_____
1 ID Identification # 1-100
1 2 Sex Sex of respondent 1=male
2=female
3 Age Age of Respondent 18-99 years
4 Fear “I fear my future 1=strongly disagree
has few 2=disagree
3=undecided
4=agree
5=strongly agree
5 Hope I have great same
hope for my future
Following is an Example of a Survey Please note the following:
There is no textbox at the beginning. The information often provided in the
textbox was provided in a cover letter.
There are a number of different indices You don’t have to always use Strongly
Disagree to Strongly Agree.
Some of the questions indicate that you can circle all that apply
This ensures that possible answer categories are exhaustive.
Some questions have an “other – specify” category.
This insures that possible answer categories are exhaustive.
ALWAYS DO Have others proofread your survey
and give you suggestions Pre-test your survey using
respondents similar to those in your sample
Submit your survey to IRB for approval
Let respondents know you are associated with USU
NEVER DO Send out surveys without IRB
approval
Contact Dr. Carol Albrecht Assessment Specialist USU Extension
