+ All Categories
Home > Self Improvement > DIY: Research on a shoestring budget

DIY: Research on a shoestring budget

Date post: 30-Nov-2014
Category:
Upload: j-todd-bennett
View: 2,461 times
Download: 1 times
Share this document with a friend
Description:
Presentation given at the CASE Communications, Marketing & Technology Conference in Boston on April 15, 2009.Learn the tools of the trade for do-it-yourself research for little or no money. This session will teach you how to conduct focus groups, surveys, usability tests and more.
54
DIY: Research On a Shoestring Budget J. Todd Bennett Managing Partner, decimal152
Transcript
Page 1: DIY: Research on a shoestring budget

DIY: Research On a

Shoestring Budget

J. Todd BennettManaging Partner,

decimal152

Page 2: DIY: Research on a shoestring budget

Why do research?

Justify

Improve

Monitor

Inform

Page 3: DIY: Research on a shoestring budget

This is not academic research

Page 4: DIY: Research on a shoestring budget

Quantitative vs. Qualitative

• QualitativeBest in earlier phases of project. Not sure

what you’re looking for.

• Quantitative

Best in later phases. Test hypotheses.

Page 5: DIY: Research on a shoestring budget

Some qualitative techniques

• Focus Groups• Interviews• Observations • Subjective analysis• Environmental Scans

Page 6: DIY: Research on a shoestring budget

Some quantitative techniques

• Surveys and questionnaires

• Data mining/ modeling

Page 7: DIY: Research on a shoestring budget

Mixed?

• Web analytics

• Usability testing

Page 8: DIY: Research on a shoestring budget

The qualitative helps give meaning and context to the

quantitative

Page 9: DIY: Research on a shoestring budget

Steps in the process

1. Define the problem2. Design the study 3. Collect data 4. Analyze data 5. Write report/ presentation

Page 10: DIY: Research on a shoestring budget

Lay the foundation

The problem“My boss wants me to do a survey”

The research questionsWhat do you want to know?

The methodWhich method will help you get there?

Be careful

Page 11: DIY: Research on a shoestring budget
Page 12: DIY: Research on a shoestring budget

How to…

Page 13: DIY: Research on a shoestring budget

Focus Groups

group interview with 10-12 members of a target audience (students,

faculty, alumni, etc.), conducted by a moderator

– relatively low cost– probe deeper as issues arise– results are almost immediate– should not be generalized to the greater

population

Page 14: DIY: Research on a shoestring budget

Recruiting participants

Need 8-12 participants, so recruit 20-25.

They should: – represent your target audience and be as

homogenous as possible – not have expertise in market research or marketing

Provide an incentive. Use existing groups only if necessary.

Page 15: DIY: Research on a shoestring budget

Focus group location

• Don’t need a fancy room• Free from distractions– private

– No public spaces

• Not too large or noisy• Seated in a circle or around a table• Pay attention to lighting and

temperature

Page 16: DIY: Research on a shoestring budget

Choosing a moderator

• Professionally trained, ideally

• Excellent speaking & listening skills

• Not too close to the project at hand

• Objective

Page 17: DIY: Research on a shoestring budget

Role of the moderator

• promote open discussion and to draw out people’s differences

• probe for details or move things forward- keep focused

• ensure everyone participates and gets a chance to speak

• LISTEN- do not control conversation or answer questions

Page 18: DIY: Research on a shoestring budget

Record the session

• Digital audio recording best

• Transcribe: http://www.verbalink.com/

• Have a notetaker in the room– Note the non-verbal– Back-up for audio

Page 19: DIY: Research on a shoestring budget

Lay the ground rules

• Introduction• Why are they here?• Encourage participation- no right or

wrong answers• Respectful of disagreement• Ensure confidentiality• Permission to record

Page 20: DIY: Research on a shoestring budget

Sample Introduction Script

Welcome to our session on [PROJECT]. Thank you for taking the time to meet with us and to discuss [PROJECT]. My name is [NAME] . Assisting me is [NAME].

You were selected because you [AUDIENCE]. We are particularly interested in your views because [REASON].

I’ll start by asking some questions and would like you to respond to them. There are no right or wrong answers. Please feel free to share your point of view even if it differs from what others have said. We do ask that you are all respectful of others’ opinions and give everyone a chance to share their views. We also ask that only one person speak at a time. Because we have a number of issues to discuss, I may have to cut you short and ask to move on. Please don’t be offended. We just need to keep things moving so we can finish on-time.

We want you to be as open and honest as possible, so we guarantee that everything you say will be kept confidential in the sense that your name will never be linked to your comments shared with [CLIENT]. However, we would like to record our conversation because it is difficult to write and follow the discussion at the same time and it’s important we don’t miss any of your comments. Do I have everyone’s consent to record the session? If so, let’s begin with the first question.

Page 21: DIY: Research on a shoestring budget

The questions

• 8-12 broad questions prepared– Several probing questions (prompts) for each

• Start with the broadest questions first

• Several seconds of silence is ok– Avoid the temptation to give examples

• Unexpected paths are ok

Page 22: DIY: Research on a shoestring budget

The results

• Notes/ Transcripts

• Content analysis

• Identify hot-button issues or areas for additional research

Page 23: DIY: Research on a shoestring budget

A few applications

Image studies

Design evaluations

Website audience needs assessment

Page 24: DIY: Research on a shoestring budget

Surveys

Page 25: DIY: Research on a shoestring budget

Survey Options

• Mail• Telephone• Online

– Solicited– Site intercept

• In-person– “mall intercept”– Group setting

Page 26: DIY: Research on a shoestring budget

Steps in conducting surveys

1. Decide the purpose2. Develop the research questions3. Define the target population4. Create the sample5. Write & design the survey instrument6. Pilot test the questionnaire7. Prepare letter/email/intro to distribute8. Follow-up9. Analyze and report

Page 27: DIY: Research on a shoestring budget

Purpose

• What are you trying to do/learn?– May be more than one objective

Example:

The purpose is to compare the content preferences of our various website audiences.

Page 28: DIY: Research on a shoestring budget

Research questions

The guide to what you will ask in the survey

Survey questions should always be

tied back to your research questions.

Page 29: DIY: Research on a shoestring budget

Purpose: to compare the content preferences of our various website

audiences.

Research questions:

• What content is most important to prospective students, faculty and alumni?

• What tasks are most important to prospective students, faculty and alumni?

• Do differences exist in preferences of prospective students, faculty and alumni?

Page 30: DIY: Research on a shoestring budget

Target Population

Who do you send the survey to?

– Determine your target population

– Is the entire population known or available?

– Where does that list come from?• HR, Admissions, Alumni database, SIS, purchased

lists (chamber of commerce, etc)

Page 31: DIY: Research on a shoestring budget

Sampling

How many people do you need to distribute it to?

– Experiment with online sample size calculator, confidence interval .05

– Expect 10-20% response from prospective students (or others who don’t know you). 60%+ from internal audiences

– Use your gut– how many responses would make you feel comfortable with results?

Page 32: DIY: Research on a shoestring budget

Sampling

Page 33: DIY: Research on a shoestring budget

Writing the survey

• NEVER ask a question unless you intend to do something with the info– Nice to know isn’t reason enough

• Tie each question to your research questions• Careful with language- avoid jargon, use

simple words, watch for bias and unequal comparisons

• Avoid “why” questions• Don’t ask “when” questions• No hypotheticals

Page 34: DIY: Research on a shoestring budget

Writing the survey

• Objectionable questions– Does it need to be asked?– For income or demographics, give ranges

• Behavior– Within 6 months– Ask “typically” or “usually”

• Attitude questions– Avoid middle alternative– Or follow with an intensity question: “How strongly do

you feel about…”

Page 35: DIY: Research on a shoestring budget

Attitudes, opinions, values

Use Likert or Semantic Differential scales

• Likert• Strongly agree, agree, disagree, strongly disagree

• Very happy, happy, unhappy, very unhappy

• Semantic Differential (bipolar adjectives)

• Efficient 7 6 5 4 3 2 1 Inefficient

• Fair +3, 2, 1, 0, -1, -2, -3 Unfair

Page 36: DIY: Research on a shoestring budget

Tips for using scales

• Scales usually indicate intensity of feeling or amount of something

• Do not include “don’t know” or “undecided” on Likert

• Use an even number of points, equally distant

• Avoid absolutes like “always” or “never” as end points

• Ask one question to determine if they are qualified to answer the following one– “Have you ever… if yes, …”

Page 37: DIY: Research on a shoestring budget

Building the survey

• Surveymonkey– Free for 100 responses. $19.95/ 1000

responses, $200 annually unlimited

• Does your institution have a tool?

Page 38: DIY: Research on a shoestring budget

The invite

• The purpose of the survey

• How will their responses help?

• What will they get in return?

• Ensure confidentiality

• Provide a deadline

• How to get results

• Thanks

Page 39: DIY: Research on a shoestring budget

Pilot test

• Feedback on the invite

• Test the flow (does the order make sense, is it monotonous?)

• Time to complete

• Do the response patterns make sense? Are they consistent?

• Do any questions cause confusion?

Page 40: DIY: Research on a shoestring budget

Distribution of surveys

– First follow-up in 5-7 days– Final reminder 1-2 days before close of

survey

You will get exponentially fewer responses to follow-ups after the first. Don’t harass your sample!

Page 41: DIY: Research on a shoestring budget

Analysis and reporting

• Basic frequencies usually sufficient• Cross-tabs by

audience/group/variable optional • Surveymonkey produces a report of

frequencies in nice charts• Excel also works fine. No real need

for SPSS.

Page 42: DIY: Research on a shoestring budget

Competitive analysis

WebsitesPublications

NewsPrograms

Page 43: DIY: Research on a shoestring budget

Competitive analysis

• Heuristics (rules of thumb)

– Navigation– Aesthetic, readability & design– Content– Multimedia and features– External Web presence

Page 44: DIY: Research on a shoestring budget

“Why aren’t we doing that?”

It’s easy to judge without context.

Institutions’ goals and strategies may differ.

While one might perceive a site/publication/strategy to be “better”

than another, each may be equally successful at fulfilling its intended

purpose.

Page 45: DIY: Research on a shoestring budget

Usability testing

Page 46: DIY: Research on a shoestring budget

Talk aloud method

• Develop scenarios – Task and information based

• Complete the tasks, talking aloud

• Observe paths, clicks, verbal impressions, body language– Note areas of difficulty

Page 47: DIY: Research on a shoestring budget

Data collection

Don’t need fancy software, but it helps

– Morae- fairly expensive– Silverback- $49, for Mac only– Cool service- www.userfly.com

Page 48: DIY: Research on a shoestring budget

Prototyping

• Paper or HTML

• Useful before development

• Quick, iterative– Mock-up, test, revise, test again

Page 49: DIY: Research on a shoestring budget

Prototyping

flickr.com/photos/bensauerflickr.com/photos/cesarastudillo

Page 50: DIY: Research on a shoestring budget

Card sorting• Useful for site architecture

(navigation, organization, labels)

• Looking for patterns (mental models) of how users group and organize content

• Open vs. closed card sorting

• Focuses on content, but not tasks

Page 51: DIY: Research on a shoestring budget

Card Sorting

.flickr.com/photos/yandle

Page 52: DIY: Research on a shoestring budget

5-second testswww.fivesecondtest.com

• Classic– list the things they can recall after viewing your site for 5 sec. – good for discovering the most prominent calls to action on pages

• Compare– choose a preferred interface after viewing each for 2.5 sec. – good for assessing design variations

• Sentiment– choose most and least favorite elements after viewing your site

for 5 sec. – good for identifying strengths and weaknesses in the design

Page 53: DIY: Research on a shoestring budget

Web analytics

Useful for measurement, but also for research

Do people do what they say they do?

• Look for ways the behavioral data supports/refutes what you learned in other

research

Page 54: DIY: Research on a shoestring budget

Questions?

Contact me:

[email protected]/jtoddb404-551-3915


Recommended