An Introduction to Research Methods: Needs Assessment, Surveys, Focus Groups and Personas
Prepared for the ULS Leadership Program by Luke Ferdinand, John Fudrow, Karen Calhoun and Jeff Wisniewski
20 November 2013
This work is licensed under aCreative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.
Agenda Topic TimeArrival 10:25-10:35 amIntroduction to the workshop 10:35-10:45 amNeeds assessment - Luke 10:45 am-12:00 pmBox lunches 12:00 -12:30 pmSurveys – John 12:30-1:30 pmShort break 1:30-1:40 pmFocus groups 1:40 -2:40 pmShort break 2:40 –2:50 pmPersonas research – Jeff 2:50 – 3:50 pmWrap up and get on shuttle 3:50 – 4:00 pm
2
Needs AssessmentDefining and Addressing The Needs of an Organization and Its Audience
Luke FerdinandULS Leadership Program20 November 2013
ULS Research Methods Workshop
4
Objectives Define Needs in Organization Define Needs Assessment Understand the methods and tools used in Needs
Assessment Know potential risks in Needs Assessment methods
ULS Research Methods Workshop
5
What is a Needs Assessment?"A systematic process for collecting information and making justifiable decisions" - Ryan Watkins
Ryan Watkins, 2013, “Needs = Gaps in Results”, http://ryanrwatkins.com/na/naintroduction.html,
ULS Research Methods Workshop
6
What is a need? “Needs are the differences between your current
achievements and your desired accomplishments.”
Ryan Watkins, 2013, “Needs = Gaps in Results”, http://ryanrwatkins.com/na/naintroduction.html,
ULS Research Methods Workshop
7
Needs Assessments Help Identify Data that define your needs Prioritization of your needs Criteria for implementing solutions Information necessary to justify selection of one or
more activities to improve performance
Ryan Watkins, 2013, “Needs = Gaps in Results”, http://ryanrwatkins.com/na/naessentials.html
ULS Research Methods Workshop
8
Why? “Provide a systematic process to guide decision-making in
organizations Provide justification for decisions before they are made. Scalable for any size project, time-frame, or budget. Offer a replicable model that can be applied by novices or
experts. Provide a systemic perspective for decision-makers. Allow for interdisciplinary solutions for complex problems.
“Ryan Watkins, 2013, Needs = Gaps in Results”, http://ryanrwatkins.com/na/naessentials.html,
ULS Research Methods Workshop
9
Where can we use a needs assessment?Strategic Action 9a : Ground Floor Needs Assessment (obviously) Opportunity to rethink the ground floor Many assumptions and ideas for what's best Gather information from stakeholders, relevant
resources Make an informed decision to best meet the needs of
organization and community Other opportunities?
ULS Research Methods Workshop
10
12 StepsBrought to you by the National Oceanic & Atmospheric Administrationhttp://csc.noaa.gov/needs/
ULS Research Methods Workshop
11
Needs Assessment Steps1. Confirm the Issues &
Audience2. Establish the Planning Team3. Establish the Goals and
Objectives4. Characterize Your Audience5. Conduct Information &
Literature Search6. Select Your Data Collection
methods
7. Determine Your Sampling Scheme
8. Design and Pilot the Collection Instrument
9. Gather & Report Data10. Analyze Data11. Manage Data12. Synthesize Data & Create
Report
ULS Research Methods Workshop
12
1: Confirm Issues & Audience• Summary:
• Establish purpose of assessment and summarize• Questions:
• Is this a new issue or audience for the organization?• Is there agreement up the organizational chain this issue or
audience needs to be addressed?• Risks:
• Unknown stakeholders• Communication issues• LACK OF SUPPORT
ULS Research Methods Workshop
13
2. Establish the Planning Team• All steps should be conducted with planning team• Ideally members of stakeholder groups and experts• Communication is essential• Consider resource needs: time, expertise etc.• Questions:
• Are the stakeholders new or well known?• How will geography impact planning?• Expertise within the team?
• Risks• Sense of ownership• Communication• Planning can overtake action
ULS Research Methods Workshop
14
3. Establish Goals and Objectives Summary
Identify the desired and actual levels of knowledge or skill Identify cause(s) for the lack of knowledge or skill Devise solution(s)
Questions: Are the goals widely shared by the audience? Are your objectives measurable? Will the project be considered a success if the objectives are met?
Risks: Potential to disengage by different team members who have different priorities Not knowing if goals have been achieved and to what degree Long term-support for doing needs assessments
Play Time
ULS Research Methods Workshop
16
4. Characterize Your Audience Summary
Sample size Skill & knowledge level Educational Level Attitudes and biases
Questions: How long have you worked with the audience? How much variation is there within the audience?
Risks: Assumptions can backfire Broadly generalizing and audience can lead to failure
ULS Research Methods Workshop
17
Topics for Audience Characterization Knowledge Training Tools & Techniques Benefits Attitudes & Biases Ability to Attend or Access Cultural Characteristics
ULS Research Methods Workshop
18
5. Information & Literature Search Summary
Environmental Scan Readings Surveys to address questions raised in review
Questions: Has this audience or issue been surveyed in the past? What other ways might we find useful information about this audience or issue?
Risks: Results may be outdated Redundant effort, wasted time Missing important information that may make results easier or better
ULS Research Methods Workshop
19
6. Select Your Data Collection methods Summary
Observation Personal Interviews Surveys Focus Groups
Questions: Have all methods been considered? Consider audience characteristics when drafting collection methods How much expertise is there in-house? Seek help!
Risks: Can be intrusive or upsetting Time spent designing vs time spent analyzing
ULS Research Methods Workshop
20
7. Determine Your Sampling Scheme Summary
Sample more than you think you need Questions:
What is statistically recommended? What is the population size of audience?
Risks: Likely not scientifically sound Too many or too few in sample
ULS Research Methods Workshop
21
Exercise: Develop Your GoalsDiscuss how you would approach
Audience CharacterizationData Collection MethodsSample SizeMeasure of Success
ULS Research Methods Workshop
22
8. Design and Pilot Your Collection Instrument Summary
Pilot first to identify weaknesses Questions:
How will you pilot your instrument? What kind of expertise is on your planning team? How important is statistical precision? How will data collection be standardized?
Risks: Instrument will not be clear or gather necessary data The audience will not be receptive to the survey instrument Asking too many questions may irritate the respondents
ULS Research Methods Workshop
23
9. Gather and Record Data Summary
Find means to incentivize respondents (survey prizes, etc) Ensure anonymity if this is a priority to audience Coordinate with assessment team If audience is outside of library, make use of External Communications group
Questions: Will you recruit people outside planning team to assist in data gathering? Are you getting the desired response rate?
Risks: Data will be biased Language or vocabulary issues Invalid Study
ULS Research Methods Workshop
24
10. Analyze Data Summary
Keep findings and interpretation of findings separate in reporting Questions:
Seek assistance in interpreting data if necessary Risks:
Under or over-whelming audience with report Missing trends and patterns Not accounting for possible critical barriers Letting bias slip into the process
ULS Research Methods Workshop
25
11. Manage Data Summary
This step involves determining how data will be organized and archived. The importance of this step is often not recognized until it is too late.
Questions: Will this data ever need to be referred to again? Will the raw data be stored? How will it be stored? Can/should the data be shared?
Risks: There may be unforeseen reasons that would necessitate the data being
used again Inadequate metadata
ULS Research Methods Workshop
26
12. Synthesize Data and Create Report Always address your goals and objectives in synthesis. Report must
include problems or errors with the design and the implementation of the survey. An executive summary is often helpful.
Questions: Who is the audience for the report? Did you address your objectives?
Risks: Potential to disengage by different team members who have different
priorities Not knowing if goals have been achieved and to what degree Long term-support for doing needs assessments
ULS Research Methods Workshop
27
Gather your Data/Report
ULS Research Methods Workshop
28
Thank you!Works Cited:
http://www.cscnoaa.gov/needshttp://ryanrwatkins.com/na (includes link to free NA eBook)
Additional Reading:http://en.wikipedia.org/wiki/Needs_assessment
Survey Basics and Survey MonkeyJohn FudrowULS Leadership Program20 November 2013
ULS Research Methods Workshop
30
When to Use a Survey Explore a Topic Discussion with Target Population Gather Objective Data on Subject Benchmarking of Service Levels
ULS Research Methods Workshop
31
Survey Monkey Templates Over 150 pre-made templates Variety of topics
Community (2) Customer Feedback (7) Demographics (9) Education (20) Events (7) Healthcare (14) Human Resources (18) Industry Specific (58) Just for Fun (9) Market Research (12) Non-profit (5) Political (11)
ULS Research Methods Workshop
32
Surveys We Have Applied ULS General Survey Library Instruction Survey E-book Survey Event Feedback Surveys Internal Planning
ULS Research Methods Workshop
33
Types of Survey Questions Open-Ended Closed-Ended
ULS Research Methods Workshop
34
Types of Survey Questions Open-Ended
i. Single Textbox ii. Multiple Textboxes iii. Comment/Essay Box iv. Numerical Textboxes v. Demographic (U.S. or International) vi. Date and/or Time
ULS Research Methods Workshop
35
Types of Survey Questions Close-Ended
i. Multiple Choice (Only One Answer Allowed) ii. Multiple Choice (Multiple Answers Allowed) iii. Rating iv. Ranking v. Matrix of Choices (Only One Answer Per Row) vi. Matrix of Choices (Multiple Answers Per Row) vii. Matrix of Drop-down menus
ULS Research Methods Workshop
36
Likert Scales and Ratings What is a Likert Scale?
A scaled response of a respondents feelings toward a topic based on a presented scale.
ULS Research Methods Workshop
37
Likert Scales and Ratings Clear wording of response choices. Number of choices to relevancy
5 to 7 choices allow for a consistent distribution. The third choice should be the undecided or neutral
decision. Be careful not to force ranking by the wording of the
base question.
ULS Research Methods Workshop
38
Anonymity• IRB Authorization
Service Improvement Anonymous Ratings
• Do not store data publically
ULS Research Methods Workshop
39
Survey Delivery Target your audience Multiple avenues for large samples Create Multiple links to analyze effectiveness of
delivery methods DON’T SPAM You cannot mass email
ULS Research Methods Workshop
40
Statistical Relevance and You We are not statisticians
We don’t have the resources nor time to run full analysis Our reports are not statistically relevant Our analysis is focused on report fulfillment
We don't properly sample Our audience is often “expert” library users
This alters the influence of their input We don't run via SPSS
Statistical Package for the Social Sciences
ULS Research Methods Workshop
41
Using Survey Monkey (Free Version) https://www.surveymonkey.com/ Create a free account
ULS Research Methods Workshop
42
Survey Monkey (Free Version) Limitations
10 Questions 100 Responses No Page Logic 20 Less Templates No Customization
ULS Research Methods Workshop
43
Survey Question Creation Create 2 questions you would like to ask of your
department or patrons Think about how you would use the results and write
that idea in the page description. We will look at several and discuss their efficacy.
Focus Groups: the Method and How to Use ItKaren CalhounULS Leadership Program20 November 2013
ULS Research Methods Workshop
45
Learning objectives1. Know the advantages and disadvantages of focus
groups and when to use them 2. Be able to select and recruit focus group interviewees3. Know how to prepare a focus group interview guide4. Have information about:
The roles of the facilitator and recorder How to analyze focus group interview data
ULS Research Methods Workshop
46
Focus Groups, Defined Qualitative social science research method Feature open, interactive, in-depth small group
discussion (typically 6 to 10 people), led by a trained moderator/facilitator
Participants carefully selected Evaluate participant thoughts, opinions, practices,
values, beliefs, feelings in a nonthreatening, semi-structured setting
ULS Research Methods Workshop
47
Purpose of a focus group IT IS TO:
Collect information and ideas on a pre-selected topic IT IS NOT:
To achieve consensus To answer participants’ questions (facilitator ≠ sage on stage) To solve one or more problems To make decisions To provide a forum for participants to gripe or vent frustration
(group therapy)
ULS Research Methods Workshop
48
Advantages of Focus Groups Generate insight into not
just what people think or feel, but why
Can be comparatively low cost method research method
Interactive, not one way Can be used alone or in
combination with other research methodsCC BY NC Francois Proulx
http://www.flickr.com/photos/91569742@N00/404909051/
ULS Research Methods Workshop
49
Disadvantages/Risks of Focus Groups Findings not representative of entire population of
interest Quality of results highly dependent on:
Appropriate preparation Skill of moderator Skill of recorder Skill of analyst (preparation of report)
Can be challenging to evaluate responses to open-ended questions
ULS Research Methods Workshop
50
Steps to Set Up Focus Group Interviews Decide how many focus group interviews to hold (usually from two to a
dozen) Select appropriate facility (see next slide) and time(s) for the event(s) Decide on participant incentives Recruit participants (usually 6 to 10 people in each focus
group) Prepare interview guide/script Assign moderator and recorder Conduct the focus group(s) Analyze and report results
ULS Research Methods Workshop
51
Focus Group Facility – Minimum Requirements
Comfortable room in a convenient location for participants Date and time convenient for participants (faculty or peers =
during work hours?; undergraduates = evening?) Table and chairs set up for group conversation Food, refreshments Permission to record the session Suitable means for moderator to conduct the interview (e.g., flip
charts, markers, laptop, projector, screen …) The means to record the session (notes + audio recording is one
way; another is streaming video to a separate room where notetakers/observers sit)
ULS Research Methods Workshop
52
Focus Group Facilities – High End
Text: Summary of information provided by a commercial focus group facility – a local one is Direct Feedback Inc. (Pittsburgh area):http://www.dfresearch.com/focus.htm
Facility and technical capabilities:•Video streaming•Wireless•Comfortable, well equipped respondent room•On-staff moderating•Assistance with recruiting, audio and video recording …•Client viewing room with one-way mirror and closed-circuit television
ULS Research Methods Workshop
53
Recruiting Participants – Selection Choose participants who will be comfortable with one another (but ideally,
do not know each other) – “homogeneous strangers” Potential selection criteria:
Knowledge of topic Comfortable discussing the topic with others Demographics - gender, age, etc. - if relevant to the topic Status , if relevant to the topic (role/target audience, e.g. faculty, student … or
user of service, non-user of service … ) If group members are known to each other, avoid having all members of a
clique in the same focus group if you can Also in groups known to each other, try to avoid having an employee and
his/her boss in the same group
ULS Research Methods Workshop
54
Recruiting Participants – Recruitment Process
Participants might be: Nominated Randomly selected Members of a definable group Have same job/title Volunteers who fit selection criteria Other
Screen potential participants/assign to groups
Once you have grouped recruits, confirm participation (see sample letter, next slide)
Send reminder 2 days before Over-invite by 10 to 20% (to account for
no-shows)
Sample flyer recruiting volunteers
Extract from Eliot and Associates 2005
ULS Research Methods Workshop
55
Sample confirmation letter Track invitation emails and responses
(along with demographic info you may need later)
Gain consent to record session (“Although the session will be recorded, your responses will remain anonymous and no names will be mentioned in the report.”)
Ideas for incentives (besides refreshments): Monetary Coupons/gift cards Door prize (drawing at end of session) …? Extract from Eliot and Associates 2005
ULS Research Methods Workshop
56
Exercise #1 Situation
You’ve been given the following assignment: Conduct focus groups of the
Pitt community to examine attitudes toward, awareness and usage of library instruction services (inside and outside the library, onsite and virtual)
Exercise Working as a group at your table,
take 10 minutes to sketch out:1. How will you determine who should
be in the focus groups (i.e. the selection criteria)?
2. Once you have selected the criteria, what methods might you use to recruit participants and assign them to groups?
Report out (1-2 minutes each table)
ULS Research Methods Workshop
57
Preparing and Using an Interview Guide(Script) Prepare pre-group paperwork – consent forms, brief
demographic information1. Opening – engagement questions
Introductions, ground rules (see sample, next slide), opening question (make it an easy one; could be round robin)
2. Exploration questions (2 to 4 key questions)3. Exit/ending questions (e.g., “Of all the things we
discussed, what is the most important to you?” and/or “Have we missed anything in our discussion today”?)
ULS Research Methods Workshop
58
Sample Introduction and Ground Rules (Interview Guide)
Extract from Eliot and Associates 2005
ULS Research Methods Workshop
59
Interview Guides Use Open-Ended Questions Open-ended questions = inquiries that produce rich
data = they start conversations and keep them going “When you think about green energy, what comes to
mind?” “What do you like most about coming to the library?”
Closed-ended questions = impose answers (yes or no, or a choice from a list) Help clarify and confirm Typically used in quantitative research like surveys
ULS Research Methods Workshop
60
Other Possibilities for Designing Focus Group Interview Guides They don’t always have to be questions …
Can show a brief video/make a proposal then start asking questions
Can introduce alternatives and ask them to choose Fill in the blank Draw a picture More
ULS Research Methods Workshop
61
Exercise #2Situation
You’ve been given the following assignment: Conduct focus groups to
evaluate user reaction to the high-tech group study rooms on the second floor of Hillman
Exercise Working at your table, and using
the handout from Richard Krueger: Take 5 minutes to select or create 3
open-ended questions for the focus group interview guide
Take another 5 minutes to consider how you will open and close the focus group interview
Each group report out (1-2 minutes)
ULS Research Methods Workshop
62
Help for Facilitators and Notetakers
Preparation for October 11 “Birds of a Feather” EventKaren CalhounOctober 7, 2013
Available from Behind the Scenes, FY14 Planning and Budget Cte. Site
This PPT covers the roles of facilitators and recorders;See also “further reading” at the end
ULS Research Methods Workshop
63
You, Moderating the Next Focus Group
ULS Research Methods Workshop
64
Analyzing Focus Group Interview Data Is often not easy Devise a method of capturing the comments from the
notes/recordings Capture the comments Look for common categories or themes Assign categories to comments Sort comments by category or theme Synthesize Summarize findings for each category/theme Prepare and present report One method described @ Eliot and Associates 2005 (see last slide)
ULS Research Methods Workshop
65
Recommended Reading for Focus Group Designers, Facilitators, Recorders and Analysts Connaway, Lynn Silipigni, Debra Wilcox Johnson, and Susan E. Searing. 1997. “Online
Catalogs from the Users’ Perspective: The Use of Focus Group Interviews.” College & Research Libraries 58 (5): 403–420.
Dixon, Jill. 2005. “Focus Group Facilitation Guidelines”. Centre for Higher Education Quality. Monash University. http://www.uwsuper.edu/cipt/exsite/upload/Focus_Group_Guidelines.pdf
Eliot and Associates. 2005. “Guidelines for Conducting a Focus Group.” http://assessment.aas.duke.edu/documents/How_to_Conduct_a_Focus_Group.pdf
FocusGroupTips.com. 2012. “Focus Group Questionnaire Fundamentals: Basic Questions.” http://www.focusgrouptips.com/focus-group-questionnaire.html
Krueger, Richard A. 2002. “Designing and Conducting Focus Group Interviews.” http://www.eiu.edu/~ihec/Krueger-FocusGroupInterviews.pdf
Walden, Graham R. 2006. “Focus Group Interviewing in the Library Literature: A Selective Annotated Bibliography 1996-2005.” Reference Services Review 34 (2) (April 1): 222–241. doi:10.1108/00907320610669461.
PersonasJeff WisniewskiULS Leadership Program20 November 2013
ULS Research Methods Workshop
67
Road Map What’s a persona? Benefits of personas How to create personas How to use personas
ULS Research Methods Workshop
68
Personas Personas are “stand ins” or hypothetical archetypes
created to represent the primary user segments for your web site
Each persona represents a key user type that shares demographic characteristics, needs, behaviours, and environment
ULS Research Methods Workshop
69
Personas Imaginary, derived from user research Each has a name and personal details
ULS Research Methods Workshop
70
Amy (Soo-Jin)
second year graduate student in Biological Engineering
currently splits her time between class work, time in the lab, and studying from home
since much of her work is either course driven or in the lab, she does not consider herself a particularly heavy library user
tends to rely on lectures rather than library resources
uses company websites quite frequently for information on the lab products she uses and uses professional association sites for recent papers and information on developments in her field
uses the popular search engines initially to get a sense of what types of materials are out there then moves to Web of Science and Compendex for access to journal articles
ULS Research Methods Workshop
71
if she can avoid going to the library, she will
at home she always connects through remote access to get access to full articles through the databases for which the library has subscription
interested in doing exhaustive searches for journal articles on her dissertation topic
no one has shown her how to use the full breadth of the resources and functionality of e-Journal on the library web site; she has a sense there are more resources and tools than she knows about
uses ILL often to gain access to articles that she cannot access through Pitt subscriptions
ULS Research Methods Workshop
72
Customer Segments to Personas Originated in the 1930 when marketers started using
fictional characters to represent a customer segment Alan Cooper, a software developer, coined a related
term and similar practice: personas. His book The Inmates are Running the Asylum
popularized the use of personas and designing for “archetypal users”.
ULS Research Methods Workshop
73
Benefits of Personas
ULS Research Methods Workshop
74
#1 “User” Centered Design Way to have users attend all your design meetings Each persona has the weight Personas are based on and embody what we know
about our library’s web site users
Personas keeping it about the user
ULS Research Methods Workshop
75
ULS Research Methods Workshop
76
#2. Support Evidence Based Decision Making Way to harness the user research data to inform web
site development Easier to remember a persona than pages of facts and
figures: path data, survey results, interview summaries etc.
Share abstract data in a compelling and memorable way Personas
encapsulate evidence
ULS Research Methods Workshop
77
#3 Where to Put Design Effort Personas spell out what the site must do to support each personas’ goals and tasks
Personas provide focus
ULS Research Methods Workshop
78
#4. Communicate to Stakeholders in a Language Understood by Everyone Easy and fun way to communicate design decisions Keeps the focus on the user Avoid “geek” speak
Personas speak to everyone
ULS Research Methods Workshop
79
#4 Build Consensus and Commitment to the Design Communicate a common direction Reduce the need for extremely detailed specifications.
Nuances of behaviours and preferences are captured in the persona and narratives
Personas build shared vision
ULS Research Methods Workshop
80
How to Create PersonasResearch!
Environmental scan Interviews Ethnographic research
ULS Research Methods Workshop
81
Environmental Scanning Identify true peers Literature review Web search Provides a framework
ULS Research Methods Workshop
82
Ethnographic Research Gorillas in the Mist Time consuming Expensive Highly useful!
ULS Research Methods Workshop
83
Interviews Useful for creating from scratch and for local validation
of “borrowed”
ULS Research Methods Workshop
84
Finding Subjects Leverage your networks General advertising not useful
ULS Research Methods Workshop
85
Logistics Voice recorder of smartphone app Transcription Analysis
ULS Research Methods Workshop
86
Interview Intro Explain hat you’re doing and why
You’re helping us build a better website Be candid
General computer usage habits When you start your browser where is the first place you
go? Favorite sites, and why?
ULS Research Methods Workshop
87
If you need to: Find books to take on vacation Write a paper…Where’s the first place you’d go?
Information-Seeking Habits
ULS Research Methods Workshop
88
How often do you go to the library? How often do you go to the library website? When do you go to the library website? What do you
do when you’re there? (Take them to site) What immediately draws your attention? What information did you look for but not find? Is there something you looked for on the homepage
but didn’t find?
Library Questions
ULS Research Methods Workshop
89
Now What?
Create
Refine
Interview
ULS Research Methods Workshop
90
How Many Personas ? Primary constituencies 5-7 generally recommended
ULS Research Methods Workshop
91
How To Use the Personas In the room Frame discussions
ULS Research Methods Workshop
92
ExerciseIn groups, create a persona based on one of the group members
Scenario: the personas created will be used to guide a user centered redesign of the ULS’ SharePoint site
ULS Research Methods Workshop
93
Questions?
ULS Research Methods Workshop
94
Thanks for coming!
Please fill out the post-workshop evaluation survey at:https://www.surveymonkey.com/s/FS8PFXK