©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Research Methods inHuman-Computer Interaction

Chapter 11-Analyzing

Qualitative Data

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Overview

• Introduction• Stages of qualitative analysis• Grounded theory• Content analysis• Analyzing text content• Analyzing multimedia content

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Stages of qualitative analysis

• Identify components of the substance• Study properties and dimensions of each

component• Understand and make inference about the

substance

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Grounded theory

• An inductive research method

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Procedures of grounded theory

• open coding• development of concepts• grouping concepts into categories• formation of a theory

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Grounded theory

• Advantages– a systematic approach to analyzing qualitative,

mostly text-based, data,– generating theory out of qualitative data that can

be backed up by ample evidence of the coding– Interplay between data collection and analysis

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Grounded theory

• Disadvantages– Researcher can be overwhelmed by the details of

the data– The theory generated is hard to evaluate– Findings may be subject to bias

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Content analysis

• A more specific view: a systematic, replicable technique for compressing many words of text into fewer content categories based on explicit rules of coding

• A broader view: any technique for making inferences by objectively and systematically identifying specified characteristics of messages

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Content categories

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Preparing for content analysis

• Define the data set• Define the population• Clean up the data• Understand the context of the data

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Analyzing text data

• A priori coding– Identify coding categories– Coding– Reliability check

• Emergent coding– Multiple coders identify coding categories based on subset of data– Consolidate category list– Code a subset of data– Reliability check– Repeat the process until satisfactory result is met– Code the rest of the data

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Identify coding categories

• Theoretical framework• Researcher-denoted concepts• In-vivo codes• Building a code structure (nomenclature)

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Coding the text

• Look for key items

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Coding the text

• Ask questions about the data• Making comparisons

– Between different coding category– Between different participant group– Between existing data and previous literature

• Using computer software

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Ensure high quality analysis

• Validity– constructing a multi-faceted argument in favor of

your interpretation of the data– Constructing a database– Data source triangulation– Interpretation should account for as much as

possible of the data– Alternative interpretations may also help

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Ensure high quality analysis

• Reliability check– Stability

• also called intra-coder reliability• examines whether the same coder rates the data in the

same way throughout the coding process

– Reproducibility• also called inter-coder reliability or investigator

triangulation • examines whether different coders code the same data

in a consistent way

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Reliability measures

• Percent agreement

• Cohen’s Kappa:

K = (Pa − Pc)/(1 − Pc)

% Agreement =

Number of cases coded the same way

Total number of cases

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Reliability check

• Agreement matrix

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Reliability check

• Interpretation of Cohen’s Kappa

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Subjective vs. objective coder• Subjective coders

– Knowledge and experience can help interpret the data

– Less training required– May cause inflated reliability

• Objective coders– Less likely to cause inflated reliability– Lack of knowledge affect the ability to understand

the data– More training required

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

Analyzing multimedia content• The supporting techniques and methods are less mature

compared to text data analysis• Approaches:

– Manual analysis: • highly labor intensive and time consuming• More accurate

– Completely automated analysis: • Faster, less amount of work• Highly inaccurate

– Partially automated approach• Combines the advantages of the manual process and the

completely automated process

©2010 John Wiley and Sons www.wileyeurope.com/college/lazar Chapter 11

End-of-chapter

• Summary• Discussion questions• Research design exercise