UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
UX ResearchQuestionnaire Analysis
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Qualitative Quantitative
•describes the qualities or characteristics of something
•provide details about human behavior, emotion, and personality characteristics
•helps to understand why people do the things they do
•any research that can be measured
•provide data that can be expressed in numbers
•answers questions like “How many people clicked here”
Reference: http://www.uxmatters.com/mt/archives/2012/09/strengths-and-weaknesses-of-quantitative-and-qualitative-research.php
ApproachesQuestionnaire
What you getLarge amount of Text
What you getNumbers
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
QualitativeQuestionnaire
HSA UX-Lab; Questionnaire 2015; Text Answers
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
QualitativeQuestionnaire
IA6-2018; Team „Grabpflege“; UX-Observation Test and Interview
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Qualitative
Reference: https://www.nsf.gov/pubs/1997/nsf97153/chap_4.htm
Questionnaire
Goal
Interpreting meaningful patterns or themes
Meaningfulness is determined by goals and objectives of the project depends on the particular research or evaluation questions
Good qualitative analysis is both: systematic and disciplined
How to analyze
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Qualitative
Reference: https://www.wikihow.com/Analyze-Qualitative-Data
QuestionnaireHow to analyzeIt's easy to be overwhelmed with the amount of content you have created.
Thematic / content analysis Generating ‘codes’ that describe themes in the text, such as ‘Anxiety’ or ‘Eating habits’.
One recommended method for Qualitative analysis to structure it:
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Qualitative
Reference: https://www.nsf.gov/pubs/1997/nsf97153/chap_4.htm
QuestionnaireHow to analyze
• What patterns and common themes emerge in responses dealing with specific items?
• Are there any deviations (Abweichungen) from these patterns?
• What interesting stories emerge from the responses? How can these stories help to illuminate the broader study question(s)?
• Do any of these patterns or findings suggest that additional data may need to be collected? Do any of the study questions need to be revised?
Ask the following questions:
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Qualitative
Reference: https://www.nsf.gov/pubs/1997/nsf97153/chap_4.htm
Questionnaire
Data Reduction
General steps
Data DisplayConclusion Drawing / Verification
Identifying Patterns and Themes
Procedure
Record and Process Data
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
QualitativeQuestionnaireProcedure Record and Process Data
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
QualitativeQuestionnaireProcedure
• Things that stuck out to you • Time/date details • Other observations • Highlights from the interaction
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-qualitative-data/
Record and Process Data As soon as data is collectedrecord detailed notes:
Do this while the interaction is still fresh in your mind so that you can record your thoughts and reactions as accurately as possible.
Make a reflection sheet template that you carry with you and complete after each interaction so that it is standardized across all data collection points.
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
QualitativeQuestionnaireProcedureUndergo a data reduction process
Determine what is significant and transform the data into a simplified format
To determine what is meaningful data always refer back to your research questions.
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-qualitative-data/
Data Reduction
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
QualitativeQuestionnaireProcedure Data Reduction
UX-Test IAM 2015 Virtuelle Architektur
„ArchiVision“
•chronological sequence of events observed •complemented by remarks of the observers • in Tabular form!
Written Log (Beobachtungsprotokoll)
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Qualitative
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-qualitative-data/
QuestionnaireProcedure
1.Coding the data for certain words or content 2. Identifying their patterns 3. Interpreting their meanings.
Content analysis:
Go through all of the text and label words, phrases, and sections of text that relate to your research questions. Now sort and examine the data to look for patterns.
Identifying Patterns and ThemesGroup data into patterns and themes by:
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
QualitativeQuestionnaireProcedure Identifying Patterns and Themes
Themes may be:
Thematic analysis
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-qualitative-data/
Group data into patterns and themes by:
Group the data into themes that will help answer the research question(s).
• Directly evolved from the research questions and were pre-set before data collection even began, or
• Naturally emerged from the data as the study was conducted.
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
QualitativeQuestionnaireProcedure Identifying Patterns and Themes
Label and assign Themes (Indexing)
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
QualitativeQuestionnaireProcedure
Provide "an organized, compressed assembly of information that permits conclusion drawing
Data Display
It should help to think about the data in new ways and assist you in identifying systematic patterns.
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-qualitative-data/
The display can be a graphic, table/matrix, or textual display.
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
QualitativeQuestionnaireProcedure
Index Tables
Data DisplayThe display can be a graphic, table/matrix, or textual display.
UX-Test IAM 2015 Virtuelle Architektur „ArchiVision“
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
QualitativeQuestionnaireProcedureConclusion Drawing / Verification
To draw reasonable conclusions:
• Step back and interpret what your findings mean • Determine how your findings help answer the research question(s) • Draw implications from your findings
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-qualitative-data/
To verify these conclusions, revisit the data (multiple times) to confirm the conclusions that you have drawn.
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
QualitativeQuestionnaireProcedure Conclusion Drawing / Verification
Where did it appear?
Description
Possible Causes
UX-Test IMS.mobile 2013Digitaler AusstellungsbegleiterBritta Diehm, Xiaomeng Jiang, Yue Ma, Kerstin Vierthaler
Issue tableUX - Observation
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
QualitativeQuestionnaireProcedure
Involve more than one person into the analytic process to serve as a cross-check and source of new ideas
Recommendations
Leave enough time for analysis and writing:Analyzing qualitative data almost always takes more time, thought, and effort than anticipated.
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-qualitative-data/
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire
Quantitative•any research that can be measured
•provide data that can be expressed in numbers
•answers questions like “How many people clicked here”
What you getNumbers
Reference: http://www.uxmatters.com/mt/archives/2012/09/strengths-and-weaknesses-of-quantitative-and-qualitative-research.php
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
HSA UX-Lab 2013, Measured Results
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
Provides quantifiable and easy to understand results.
Data may have one of the following formats
Nominal (Nennwert)Ordinal (Ordnungszahl)Interval (standardisierter Abstand)Ratio (Scale) (Verhältnis)
Levels of measurement
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire
Reference: https://www.youtube.com/watch?v=klgFMJppfcY
Levels of measurement
J David Eisenberg
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
Levels of measurementNominal (Nennwert)
Example:
Male or Female There is no order associated with male nor female Each category is assigned an arbitrary value male = 0, female = 1
no logical orderbasic classification data
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
Levels of measurementOrdinal (Ordnungszahl)
Example:
T-shirt size (small, medium, large) Military rank (from Private (Gefreiter) to General)
Data has a logical orderDifferences between values are not constant
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
Levels of measurementInterval (standardisierter Abstand)Data is continuous and has a logical orderhas standardized differences between values, but no natural zero
Example: Items measured on a Likert scalerank your satisfaction on scale of 1-5.
1 = Very Dissatisfied 2 = Dissatisfied 3 = Neutral 4 = Satisfied 5 = Very satisfied
Example: Fahrenheit or Celsius degrees Remember that ratios (Relationen) are meaningless for interval data. You cannot say, for example, that one day is twice as hot as another day.
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
Levels of measurementRatio (Scale) (Verhältnis)
Data is • continuous • ordered • has standardized differences between values • and a natural zero
Example:height, weight, age, length Having an absolute zero enables you to meaningful say that one measure is twice as long as another.10 cm is twice as long as 5 cm
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire
Reference: https://www.youtube.com/watch?v=eghn__C7JLQ
Levels of measurement
365 Data Science
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
• Data tabulation • Descriptive analysis • Data disaggregation • Analysis of Variance
Analysis procedures
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
Example Study (fictional)Summer program student surveyconducted with
Some standard methods
How to analyze
22 participants in two cities (NYC, Boston)
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire QuantitativeAnalysis procedures
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
Organize your collected data in a logical format Use a spreadsheet software (Libre Office Calc)
22 participants in two cities (NYC, Boston)
Some of the responses are entered as numerical values.
Each participant is assigned to a unique participant ID responses are organized by survey item/question.
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
QuantitativeQuestionnaireAnalysis procedures
Some responses transformed intonumerical values
• variable/question name • the answer options • the numerical code
assigned to answer option
Create and Keepa Code book that lists:
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire QuantitativeAnalysis procedures Data tabulation (Tabellierung)
Tabulate your results in your data set Construct frequency and percent distributions
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John?
Questionnaire QuantitativeAnalysis procedures Descriptive analysis
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
•Mean the numerical average for a particular variable
•Minimum and maximumthe highest and lowest value for a particular variable
•Mode most common number score or value for a variable
Your Results
For a first interpretationdetermine the
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire QuantitativeAnalysis proceduresMean (Arithmetisches Mittel, Durchschnitt)
Numerical average for a particular variable
Obtained by summing all answers or scores and dividing by the total number of entries
Reference: http://learningstore.uwex.edu/assets/pdfs/G3658-6.pdf
Descriptive analysis
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire QuantitativeAnalysis proceduresMean (Arithmetisches Mittel, Durchschnitt)
Numerical average for a particular variable
Obtained by summing all answers or scores and dividing by the total number of entries
Reference: http://learningstore.uwex.edu/assets/pdfs/G3658-6.pdf
Descriptive analysis
?Example
The wait times (in minutes) of five customers in a bank: 3, 2, 4, 1, and 2.
(3 + 2 + 4 + 1 + 2) : 5 = 12 : 5 = 2,4 min.
The mean waiting time:
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire QuantitativeAnalysis proceduresMean (Arithmetisches Mittel, Durchschnitt)
Descriptive analysis
https://support.minitab.com/en-us/minitab/18/help-and-how-to/statistics/basic-statistics/how-to/store-descriptive-statistics/interpret-the-statistics/interpret-the-statistics/#mean
Use the mean to describe the sample with a single value that represents the center of the data.
Many statistical analyses use the mean as a standard measure of the center of the distribution of the data.
Example
The wait times (in minutes) of five customers in a bank are: 3, 2, 4, 1, and 2.
(3 + 2 + 4 + 1 + 2) : 5 = 12 : 5 = 2,4 min.
The mean waiting time:
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire QuantitativeAnalysis procedures
Reference: http://learningstore.uwex.edu/assets/pdfs/G3658-6.pdf
Mode (Modalwert, Häufigster aller Werte)
Most commonly occurring answer or valueUsually what people refer as: „the typical“
Only important when a large number of values is availableNot effected by extrem values.
Descriptive analysis
ExampleThe wait times (in minutes) of five customers in a bank are: 3, 2, 4, 1, and 2.
2 min.The mode waiting time: ?
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire QuantitativeAnalysis procedures
Reference: http://learningstore.uwex.edu/assets/pdfs/G3658-6.pdf
Median (mittlerer aller Werte)
Middle Valuehalf of the cases fall below, half above the value
Arrange data from one extreme to the otherCount halfway through the list and find the median value
When two numbers tie for the halfway point,take the two middle numbers, add them and divide by 2 to get the median
Not effected by extremes or a range of data (like mode)
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Reference: http://learningstore.uwex.edu/assets/pdfs/G3658-6.pdf
Analysis proceduresMean, Mode, Median
Depends on your dataand the purpose of analysis
Which makes more sense?Durchschnitt, Häufigster, Mittlerer
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
One value drastically changed
Analysis proceduresMean, Mode, Median
$460,000
Recalculate!
Which makes more sense?Durchschnitt, Häufigster, Mittlerer
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
One value drastically changed
Analysis proceduresMean, Mode, Median
$460,000
Recalculate!
Which makes more sense?
$780,000
$78,000
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Disaggregating (Aufschlüsselung)Analysis procedures
Crosstabs (Kreuztabellen)
Disaggregate the data across multiple categories Allow to explore findings in depth Useful to show differences among subgroups
Reference: http://learningstore.uwex.edu/assets/pdfs/G3658-6.pdf
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
• Females overrepresented in New York,males overrepresented in Boston
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
Analysis procedures
• Over 70% of the White sample is in the Boston program
• Altogether 6 of the Boston respondents are students of color
Disaggregating (Aufschlüsselung)Crosstabs (Kreuztabellen)
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
All of the students who were dissatisfied with the program were students of color.
Analysis procedures
From these results it may be concluded that the Boston program is not meeting the needs of its students of color.
5 of the 20 students were dissatisfied with the program
All but one of the students of color in the Boston program were dissatisfied (since there were 6 students of color in the Boston program.)
Disaggregating (Aufschlüsselung)
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Correlation (Wechselwirkung)
Relationship between two variables (strong and negative, weak and positive, statistically significant)
• Correlation does not explain causation • Indicates that a relationship or pattern exists,
but it does not mean that one variable is the cause of the other.
You might see a strong positive correlation between participation in the summer program and students’ grades the following school year. However, the correlation will not tell you if the summer program is the reason why students’ grades were higher.
Analysis procedures
Example
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Variability (Schwankungsbreite)Analysis procedures
Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
One extreme (high or low) may have strong influence on your over all mean (Durchschnittswert)
Looking at variability provides a better understanding of the results
Analysis of variance (ANOVA)determines whether the difference in means (averages) for two groups is statistically significant.
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Reference: https://www.youtube.com/watch?v=ITf4vHhyGpc Analysis of variance (ANOVA)J David Eisenberg
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Reference: http://learningstore.uwex.edu/assets/pdfs/G3658-6.pdf
Analysis proceduresMeasures of Variability
Range
Compares the highest and the lowest valueto indicate the spread of responses or scores
Example
Soil testing for phosphorus saved producers anaverage $15 / acreranging from $12 to $20 / acre
Only considers the highest and the lowest score.Helps to better understand the date but not a full measure of variation
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire
Reference: https://www.youtube.com/watch?v=ipYaHqutMds
Measures of Variability
Lydia Flinn
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire QuantitativeAnalysis proceduresMeasures of Variability
Standard deviation (Abweichung)
Measures the degree to which individual values vary from the mean
High standard deviation means responses vary greatly from the mean.
https://www.mathsisfun.com/data/standard-deviation.html https://www.easycalculation.com/statistics/standard-deviation.php
How to calculate?Detailed but easy to understand explanation and examples at:
Why is it useful in UX?
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire QuantitativeAnalysis proceduresMeasures of Variability
Standard deviation (Abweichung)
Compare the UX of two products To evaluate differences between two versionscheck if the differences are significant
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Reference: IMS.UX WS 2017-18, HSA, Prof. Dr. Hariet Köstner
Presentation of Results
Who do you address?
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Reference: IMS.UX WS 2017-18, HSA, Prof. Dr. Hariet Köstner
Presentation of Results
suitable for
• questions with many answers • group comparisons, frequencies, percentages, means can be shown
Bar chart
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Reference: IMS.UX WS 2017-18, HSA, Prof. Dr. Hariet Köstner
Presentation of Results
• answers that sum up to 100 • not to many answers
suitable for
Pie chart
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Reference: IMS.UX WS 2017-18, HSA, Prof. Dr. Hariet Köstner
Presentation of ResultsStacking Diagram
• answers that sum up to 100 • not to many answers
suitable for
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Questionnaire Quantitative
Reference: IMS.UX WS 2017-18, HSA, Prof. Dr. Hariet Köstner
Presentation of ResultsLine chart
suitable for • questions with many answers • group comparisons, frequencies, percentages, means can be shown • compact graphical representation of a lot of information
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
PracticeAnalyse your dataand communicate your findings
based on your research question:
How do changes in design and implementation of the survey
influence its outcome?
Present (12 Min.) and deliver (PDF) your results next week
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Practice Criteria
• Precise definition ofResearch Question and measurements
• definition of biases and striven tendency
• Methodically consistent implementation and execution
• Critical Evaluation of test procedure
• Clear summery of findings
Example: IMS.UX 2018 - Team Wulfpack
UX ResearchQuestionnaires / Surveys
Analysis
KP Ludwig John
Practice Criteria
hs-augsburg.de/homes/john
Next appointments and script: