©2010 John Wiley and Sons Chapter 11 Research Methods in Human-Computer Interaction Chapter 11-...

transcript

Research Methods inHuman-Computer Interaction

Chapter 11-Analyzing

Qualitative Data

Overview

• Introduction• Stages of qualitative analysis• Grounded theory• Content analysis• Analyzing text content• Analyzing multimedia content

Stages of qualitative analysis

• Identify components of the substance• Study properties and dimensions of each

component• Understand and make inference about the

substance

Grounded theory

• An inductive research method

Procedures of grounded theory

• open coding• development of concepts• grouping concepts into categories• formation of a theory

Grounded theory

• Advantages– a systematic approach to analyzing qualitative,

mostly text-based, data,– generating theory out of qualitative data that can

be backed up by ample evidence of the coding– Interplay between data collection and analysis

Grounded theory

• Disadvantages– Researcher can be overwhelmed by the details of

the data– The theory generated is hard to evaluate– Findings may be subject to bias

Content analysis

• A more specific view: a systematic, replicable technique for compressing many words of text into fewer content categories based on explicit rules of coding

• A broader view: any technique for making inferences by objectively and systematically identifying specified characteristics of messages

Content categories

Preparing for content analysis

• Define the data set• Define the population• Clean up the data• Understand the context of the data

Analyzing text data

• A priori coding– Identify coding categories– Coding– Reliability check

• Emergent coding– Multiple coders identify coding categories based on subset of data– Consolidate category list– Code a subset of data– Reliability check– Repeat the process until satisfactory result is met– Code the rest of the data

Identify coding categories

• Theoretical framework• Researcher-denoted concepts• In-vivo codes• Building a code structure (nomenclature)

Coding the text

• Look for key items

Coding the text

• Ask questions about the data• Making comparisons

– Between different coding category– Between different participant group– Between existing data and previous literature

• Using computer software

Ensure high quality analysis

• Validity– constructing a multi-faceted argument in favor of

your interpretation of the data– Constructing a database– Data source triangulation– Interpretation should account for as much as

possible of the data– Alternative interpretations may also help

Ensure high quality analysis

• Reliability check– Stability

• also called intra-coder reliability• examines whether the same coder rates the data in the

same way throughout the coding process

– Reproducibility• also called inter-coder reliability or investigator

triangulation • examines whether different coders code the same data

in a consistent way

Reliability measures

• Percent agreement

• Cohen’s Kappa:

K = (Pa − Pc)/(1 − Pc)

% Agreement =

Number of cases coded the same way

Total number of cases

Reliability check

• Agreement matrix

Reliability check

• Interpretation of Cohen’s Kappa

Subjective vs. objective coder• Subjective coders

– Knowledge and experience can help interpret the data

– Less training required– May cause inflated reliability

• Objective coders– Less likely to cause inflated reliability– Lack of knowledge affect the ability to understand

the data– More training required

Analyzing multimedia content• The supporting techniques and methods are less mature

compared to text data analysis• Approaches:

– Manual analysis: • highly labor intensive and time consuming• More accurate

– Completely automated analysis: • Faster, less amount of work• Highly inaccurate

– Partially automated approach• Combines the advantages of the manual process and the

completely automated process

End-of-chapter

• Summary• Discussion questions• Research design exercise

©2010 John Wiley and Sons Chapter 11 Research Methods in Human-Computer Interaction Chapter 11-...

Documents