QUALITATIVE EVALUATION OVERVIEW

Kim KoesterDirector of Qualitative ResearchAIDS Policy Research CenterCenter for AIDS Prevention StudiesUCSF

Objectives

To teach you to understand the application of and methods associated with qualitative evaluation

To provide an overview a peek “under the hood” of the qualitative analytic process.

Outline of talk

1. Explain qualitative research process1. Methods2. Procedures 3. Sampling4. Analysis

2. Demystify the analytic activities 1. Briefly describe different ways to approach qualitative

data analysis.3. Dig in to practical issues

1. Talk about resources required2. Touch on issues that must be addressed in writing up

data for publication

Questions for you

What comes to mind when you think of qualitative methods?

Who plans to incorporate qualitative methods into your evaluation plan for this class?

Has anyone participated in a qualitative research project, as either an informant or as a member of a research team?

What is Qualitative Research?

Qualitative research is an approach to scientific inquiry that relies on more

naturalistic, humanistic and interactive processes. The methods are primarily

language based, with data in the form of words rather than numbers.

Joan S. Ash and Kenneth P. Guappone 2007

Qualitative Inquiry Areas of Emphasis Emphasizes a holistic perspective

Strive to understand a program and situation as a whole

Emphasizes the importance of understanding the meanings of human behavior and the social-cultural context of social interaction.

Emphasizes the emic perspective

Qualitative researchers empathize and identify with

the people they study in order to understand how those

people see things. Taylor and Bogdan

Qualitative Methods

Geared towards exploration, discovery, inductive logic

Defined in comparison to deductive research that begins with an hypothesis.

Comparing Qualitative & Quantitative Methods

Inductive Small n Purposeful sample Open-ended

questions, geared towards discovery

Seeking holistic understanding

Not generalizable Investigates how or

why

Deductive Large n Random sample Closed-ended

questions, predetermined measures

Individual variables Generalizable Investigates how

many when, where

Qualitative Techniques

In-depth interviewing (IDI) Key informant interviews Semi-structured interviews Narrative interviews

Life history interviews Sexual history interviews Critical incident interviews

Focus groups or Discussion groups Ethnography

Participant observation – shadowing/tag alongs

Selecting the method

The method you select depends entirely on the research question.

Give me an example and we’ll talk through the best method.

What methods used in the articles you read for today?

Sampling

Sample size in qualitative research vary from project to project; “there are no rules for sample size”

Purposive or purposeful sampling is the most common strategy in applied research/evaluation.

Purposeful sampling relies on the selection of information-rich cases; information rich cases allow learning a great deal about the issues that are of central importance to the evaluation

Patton delineated 16 different types of sampling frames

Designing Interview Guides

The purpose of the interview guide is to list the questions or areas of inquiry to be approached in the interview setting. Semi-structured interviews In-depth interviews

Example Interview Guide

DOMAIN: PERSONAL HISTORY/LIFE CIRCUMSTANCES INTENT: The intent of this section is to learn about

what has happened since the person was released and about how they are doing, in general.

 QUESTIONS:Tell me how you have been meeting your day-to-day survival

needs, for example, housing, money, food, etc.?Tell me about your experiences re-connecting with friends and

family since you got out of prison.

 Qualitative Analysis

Qualitative analysis is the process by which you turn your data into findings.

What are you doing when you are analyzing data? Reducing the sheer volume of raw data Separating out trivia from significance Identifying significant patterns

Qualitative Analysis

Driven by goals and aims of the evaluation Describe program Elucidate program dynamics Identify patterns Confirm or disconfirm quantitative findings

Key analytic activities

Thinking Reading Noticing Reacting Learning Producing insights Reading Sharing & discussing

•Refining insights•Reading •Labeling •Producing insights •Labeling•Reading•Parsing•Assembling•Understanding•Interpreting •Reading •And so on

Phenomenology

What is the meaning, structure, and essence of the lived experience of this phenomenon for this person or group of people?

Goal: to elucidate meaning of lived experience

Narrative Analysis

What does this narrative or story reveal about the person and world from which it came? How can this narrative be interpreted so that it provides an understanding of and illuminates the life and culture that created it? Comparing stories and cases to create meaning

Goal: to interpret stories to reveal cultural patterns

Grounded Theory

What theory emerges from a systematic comparative analysis and is grounded in fieldwork so as to explain what has been and is observed? Theoretical sampling Theoretical saturation Open and axial coding

Goal: to generate explanatory theory from social processes

Best uses: unstructured in-depth interviews, structured data sets with rich, lengthy narratives

Framework Analysis

Commonly used in applied policy research, this approach involves a set of distinct, yet highly interconnected analytic stages.

Familiarization w/data Identify thematic framework Indexing or coding Charting Mapping and Interpretation

Goal: to follow a transparent analytic process Best uses: interdisciplinary, team-based

research, in depth interviews and focus group data

Content Analysis

• Involves tagging a set of texts or other artifacts with codes that are derived from theory or from prior knowledge and then analyzing the distribution of the codes, usually statistically

• Once codes are developed, they become the unit of analysis rather than the utterances to which codes are assigned

• Goal: begin with deductive application of a priori codes, but remain open to inductive revelations.

• Best uses: Structured in-depth interviews, focus group

Software

Qualitative software packages exist to facilitate the organization of qualitative data.

Dedoose single user $(15 per month)

Atlas.ti single user $1800 Multi-user (5) - $6700

Nvivo single user - $1300 Multi-user (5) - $5200

Simply observing and interviewing do not ensure that the research is qualitative; the qualitative researcher must also interpret the beliefs and behaviors of participants.

Valerie Janesick

Disposition Ideas

Characteristics of analysts: Significant amount of patience and

tolerance for reading and re-reading data Able to sit with ambiguity Non-linear orientation Creative