Excellence in Qualitative Research

Kathleen A. Knafl, Kathleen A. Knafl, PhD, FAANPhD, FAAN

Associate Dean for ResearchFrances Hill Fox Distinguished Professor

The University of North Carolina at Chapel Hill

Learning to Evaluate Qualitative Research

Striving for quality as a researcher

Being evaluated – successful/unsuccessful attempts to publish and secure funding

Evaluating others – proposal and manuscript reviewer

Overview of Presentation

Nature and purpose of qualitative research

Position statements and guidelines for evaluating qualitative research

Common expectations and indicators of excellence

Overall Goal

Avoid an overly simplified or overly complex discussion of issues

Present guidelines that address scientific rigor while recognizing the distinct qualities and contributions of qualitative research

“Creativity must be preserved within qualitative research, but not at the expense of the quality of the science” (Maxwell, 1990)

Nature and Purpose of Qualitative Research

Multiple approaches

Terminological jungle Ethnography

Phenomenology

Hermeneutics

Constructivism

Participatory action research

Grounded theory, etc…

Common Characteristics of Qualitative Research

Focus on subjective perspective of respondents

Emphasis on contextual understanding

Design flexibility; emergent design

Generation of narrative data

Researcher as primary data collection instrument

Differences Across Approaches

Purpose

Philosophical underpinnings

Disciplinary roots

Theoretical orientation

Dominant methods for data collection and analysis

Varying Purposes of Qualitative Research

Description

Conceptualization and theoretical understanding

Empowerment and social change

Description (e.g. descriptive phenomenology, ethnography, qualitative description)

“The purpose of this study is to exploreexplore how spousal carers of people with MS interpreted their lived experience lived experience with their partner, the way in they assigned meaningmeaning to their being in such a situation, and the skills and knowledge they have developed to live with their situation” (Cheung & Hocking, 2004, p. 155).

Conceptualization (e.g. grounded theory, concept development)

“The author investigated decision-making experiences of 20 surrogates who assisted terminally ill family members for this grounded theory study. Findings describe a basic social process basic social process of Seeing them Through with Care and Respect” (Meeker, 2004, p.204).

Empowerment and Change(e.g., feminist, participatory action research)

“In this participatory research study, injured injured

workers and academics togetherworkers and academics together participated in

setting the agenda, determining the research

questions and methods, gathering the data, and

interpreting the results. The process helped all

understand more deeply the complicated reality of

being an injured worker within a set of system

and societal norms (Beardwood, et al., 2005, 33-344).

The Challenge of Evaluating Qualitative Research Evaluative stance – Should the same

criteria apply to all research? All qualitative research?

Reasonable expectations – What should we expect to see included in a qualitative report/proposal?

Appropriate criteria – What standards should we use to judge the merits of a qualitative report/proposal?

Guidelines and Criteria

Guidelines for reporting (Tong, et.all, 2007)

Shared standards (Collingridge & Gantt, 2008; Guba & Lincoln, 1985)

Alternative standards (Cheek, 2007; Cohen & Crabtree, 2008; Davies & Dodd, 2002; Fossey, et al., 2002; Whittemore, et al., 2001)

Shared Standards (Lincoln & Guba, 1985)

Truth Value (internal validity/credibilitycredibility)

Applicability (external validity/transferabilitytransferability)

Consistency (reliability/auditabilityauditability)

Neutrality (objectivity/confirmabilityconfirmability)

Qualitative standard Qualitative standard

Credibility (internal validity)

Defined as adequate representation of group or situation studied

Quality of the data

Demonstrated by prolonged engagement, persistent observation, triangulation, external checks, negative case analysis, member checking

Transferability (external validity)

Defined as the degree of similarity between research site/participants and others

Demonstrated by thick description, reporting of information for judgments about transferability

Joint responsibility of researcher and consumer

Dependability (reliability)

Defined as differentiating real perspective/behavior of respondents from reactive effects of research.

Demonstrated by audit of data collection processes Quality of interaction with subjects

External review of data

Confirmability (objectivity)

Defined as the adequacy of the results, interpretations, and recommendations

Demonstrated by audit of analytic procedures Data processing and reduction

Data review and interpretation

External auditor

Alternative Standards (Davies & Dodd)

“Incorporating the notion of ethics, the authors develop a cluster of terms around which they argue that qualitative research can meaningfully speak about rigor: attentiveness, empathy, carefulness, attentiveness, empathy, carefulness, sensitivity, respect, reflection, sensitivity, respect, reflection, conscientiousness, engagement, awareness conscientiousness, engagement, awareness and opennessand openness” (Davies & Dodd, 2002, p. 279)

Common Evaluative Criteria (Cohen & Crabtree, 2008; Tong, et al., 2007; Whittemore, 2001)

Importance of the research

Use of appropriate methods

Coherence of the research

Clarity of the presentation

Ethical conduct of research

Application of Common Criteria to Research Reports/Proposals

Reasonable expectations

Indicators of excellence

Importance of Research

Reasonable expectations General statement of purpose, research

questions, or aims

Review of relevant literature

Importance of Research

Indicators of excellence Evidence of building on prior research;

addressing an important gap

Pragmatically or theoretically useful

Focus on the research problem, not the method

Making the Case for Importance

“The aim of this study was to identify approaches

primary care providers use to engage older men

in depression care. We focus specifically on

older men because previous work has shown

that they suffer from higher levels of under-they suffer from higher levels of under-

treatmenttreatment and that men differ from women in

gender-specific barriers to help seeking”

(Apesoa-Varano, et al., 2010, p. 587)

Use of Appropriate Methods – Study Design

Reasonable expectations Varied approaches

Emergent design

Indicators of excellence Rationale for selection of approach;

fit with study purpose

Citations in support of the approach

Rationale for Study Design

“We chose grounded theory grounded theory as the method for

data collection and analysis. It is a method for

conceptualizing patterns of behavior patterns of behavior in which

people are engaged (Glaser,1978;1998). In

this study, the patterns of behavior are those in

which nurses engage while caring for palliative

cancer patients in hospitals” (Sandgren, et al., 2006,

p. 80).

Use of Appropriate Methods – Role of Theory (Sandelowski, 2003)

Reasonable expectations Variation in the source, function, and temporal

placement of theory

Different qualitative traditions specify different roles of theory

Prevailing (though misleading) belief that qualitative research is atheoretical

Use of Appropriate Methods – Role of Theory

Indicators of excellence Explicit statement of philosophical beliefs,

concepts, or theories that informed design of study

When a concept/theory is an outcome of the research, comparison to related concepts or theories

Example - Use of Theory

“The philosophical framework philosophical framework for this study was based on interpretive phenomenology” (Cheung, 2004, 155).

“The purpose of this study was to examine older African American women’s perceptions of social and environmental stress in relation to their heart disease through the lens of the through the lens of the weathering conceptual frameworkweathering conceptual framework” (Warren-Findlow, 2007, 234).

Sample Size and Selection

Reasonable expectations Purposive approach; non-probability sample

Relatively small sample

Single case to 50+ participants

Likely range of number of participants

Sample Size and Selection

Indicators of excellence Statement of purposive intent (e.g. maximum

variation, theoretical, intensity)

Specification of unit of interest (individual, group, setting)

Rationale for sample size; invoking the principle of saturation

Purposive Sample – Maximum Variation

“The base line criterion for inclusion in the sample was that all parents were employed and had at least one child whom they identified as disabled. The aim was to generate a sample that The aim was to generate a sample that included a range of situations in which parents included a range of situations in which parents combined employment and care. combined employment and care. This was not intended to achieve generalizability, but to enable examination of issues not related to a specific homogenous group” (Lewis, et. al, 2000,

p.1035).

Purposive Sample – Theoretical

“ As data collection progressedAs data collection progressed, I conducted theoretical sampling to provide data needed to describe the categories thoroughly. For example, because early participants were all reporting that other family members had been very supportive, I sought participants who had experienced conflict (Meeker, 2004,

p.208) .

Data Collection – Interviewing and Observation

Reasonable expectations Use of single or multiple data collection

strategies

Description of:• Setting for data collection

• Duration of data collection

• Identification of data collectors

• Description of observational or interview guides

• Method for recording data

Data Collection – Interviewing and Observation

Indicators of excellence Overall explicitness and thoroughness

• Description of role of the investigators; nature of interactions with the participants

• Options for follow-up contact with participants

• Efforts to assure and monitor data quality

• Evidence of sensitivity to incoming data; emergent design

Example – Interviews

Each participant was told, “I’d like you to tell me

the storystory of your experience with a chronic health

condition. Start at the time your symptoms began

and describe the things that happened one after

another regarding your health condition until

today. I encourage details because whatever is

important to you is of interest to me” (Lee & Poole,

2005, 349).

Example - Observation

“I collected data through participant observation in two cardiac rehabilitation programs. In both, I assisted in the day-to-day work of the staff, I assisted in the day-to-day work of the staff, attended education sessions, and conversed attended education sessions, and conversed with participants on an ongoing basiswith participants on an ongoing basis. When I introduced myself to clients, I indicated I was there to learn about heart disease from the participants’ point of view and that I was carrying out research (Wheatley, 2005, p. 440-441).

Data Processing and Analysis

Reasonable expectations Overview of the steps the researcher took to

break the data into smaller units for the purpose of analysis

Methodological citations that support the approach

Variability in use of software programs to support analysis

Data Processing and Analysis

Indicators of excellence Analytic process is consistent with the

qualitative approach • Line by line coding for grounded theory

• Extraction of significant statements for phenomenology

The unit of study is preserved in the analysis

Evidence of thoroughness and checks on quality

Explicitness

Results

Reasonable expectations Variation in organizing structure for

presenting results• Process

• Essential structure

• Thematic description

• Typology

Use of illustrative quotes or vignettes

Variable use of numbers

Results

Indicators of excellence Vividness

• Compelling presentation

• A good read

• Creative

Coherence• Integration of the data; more than a description

of codes and themes

• Convincingly addresses all study aims

Discussion

Reasonable expectations Possibly integrated with results

Linkages to the body of knowledge in the field

Statement of applied and/or theoretical implications of the findings

Limitations and next steps

Discussion

Indicators of excellence Elaboration of pragmatic/theoretical implications

Limitations that address the unique aspects of qualitative research (e.g. non-probability sample not a limitation)

Explicit consideration of transferability

Ethics

Reasonable expectations Statement that IRB approval has been obtained

Indicators of excellence Evidence of sensitivity to human, social, and

cultural contexts

Recognition of ongoing ethical issues and decisions

Overall Coherence and Consistency

Alignment across all aspects of the study

Completeness and follow through

Explicitness; thoroughness of presentation

Creativity – thoughtful, innovative use of methods; insightful linkages

Balancing Rigor and Creativity

“We can preserve or kill the spirit of qualitative work; we can soften our notion of rigor to include playfulness, soulfulness, imagination, and technique we associate with more artistic endeavors, or we can further harden it by the uncritical application of rules. The choice is ours rigor or rigor mortis” (Sandelowski, 1993).

The University of North Carolina at Chapel Hill