On evidence in HTA: Bringing patient, caregiver and community experience in Taking account of what can’t be counted: The place of qualitative evidence in HTA

Fiona A. Miller, PhD @FionaAliceMill

Associate Professor, IHPME, THETA

CADTH Symposium

April 11, 2016, Ottawa

2

What is evidence?

• Origins (Upshur, 2001)

– Middle English: via Old French from Latin evidentia, from evident- 'obvious to the eye or mind' (Oxford)

• Definitions (Upshur, 2001)

– something that makes plain or clear; an indication or sign (Dictionary.com)

– something that furnishes proof (Merriam-Webster)

• Facts alone are not evidence (Madjar & Walton, 2001)

– “facts can only become evidence in response to some particular question” (Chandler et al, 1994)

3

What questions might HTA need to answer?

Describing/ Characterizing

• Patients

– The meaning and experience of illness

• Technologies

– The meaning and experience of current and proposed technologies

• Services/ Systems

– The arrangement and operation of current and proposed services

Interrogating/ Explaining • Patients

– How social, cultural and economic contexts condition illness

• Technologies – How social, cultural and

economic contexts condition the effects of technology

• Services/ Systems – How services and systems

condition experiences and outcomes

– What makes some interventions work better than others

4

Answering “other” questions: The need for “other” evidence

• From and about persons

– ‘knowledge in the possession of people’ (JBI, 2014)

– Physically, socially, culturally embodied and embedded

• Only partially captured in the clinical and economic evidence traditionally used in HTA

5

Sources of “other” evidence

Research-based

• Often “qualitative”

– Interviews, focus groups

– Ethnographic observation

– Questionnaires, documents

• Descriptive

• Interpretive

• Theory

Opinion-based

• Patient experts - Expert opinion

– Input

– Participation

– Deliberative engagement

• Descriptive

• Interpretive

• Judgment

6

Often seeks to “give voice”

May use research-based methods

ON RELEVANCE AND RIGOUR

The quality of “other” evidence

7

Relevance

• On relevance – Being ‘pertinent to…’, ‘bearing upon…’, ‘connected with…’,

or ‘appropriate to…’, ‘…the matter at hand’, as well as ‘germane’, ‘apropos’, ‘material’, and ‘applicable’ (Dobrow et al, 2015)

• Opinion has a priori relevance – Local, specific

• (And other functions beyond the probative)

• Research has complex relevance – Aggregative function – persistence of themes

– Configurative function – unanticipated connections between themes (Sandelowski et al, 2012)

– Comparative function – the contingency of current practice 8

Rigour

• On rigour

– The quality of being extremely thorough, exhaustive, or accurate … meticulous, thorough, careful, diligent, scrupulous, exact, precise, correct

• A function of act and actor

– Appropriate

– Expert

– Trustworthy

9

Critical appraisal tools

• QARI, JBI

– Critical appraisal of qualitative evidence

• NOTARI, JBI

– Critical appraisal of text and opinion based evidence

10

QARI Critical Appraisal Criteria (JBI, 2014)

• 1. Congruity between the stated philosophical perspective and the research methodology

• 2. Congruity between the research methodology and the research question or objectives

• 3. Congruity between the research methodology and the methods used to collect data

• 4. Congruity between the research methodology and the representation and analysis of data

• 5. There is congruence between the research methodology and the interpretation of results

• 6. Locating the researcher culturally or theoretically • 7. Influence of the researcher on the research, and vice-versa, is

addressed • 8. Representation of participants and their voices • 9. Ethical approval by an appropriate body • 10. Relationship of conclusions to analysis, or interpretation of the

data 11

NOTARI Critical Appraisal Criteria (JBI, 2014)

• 1. Is the source of opinion clearly identified? • 2. Does the source of opinion have standing in the field

of expertise? • 3. Are the interests of patients/clients the central focus

of the opinion? • 4. Is the opinion’s basis in logic/experience clearly

argued? • 5. Is the argument that has been developed analytical?

Is the opinion the result of an analytical process drawing on experience or the literature?

• 6. Is there reference to the extant literature/evidence and any incongruence with it logically defended?

• 7. Is the opinion supported by peers?

12

Rigour

• On rigour

– The quality of being extremely thorough, exhaustive, or accurate … meticulous, thorough, careful, diligent, scrupulous, exact, precise, correct

• A function of evidence systems (Pang et al, 2003)

– Stewardship

– Financing: securing and allocating funds

– Creating and sustaining resources

– Producing, utilizing and synthesizing evidence

13

Health “evidence” systems (Pang et al, 2003)

14

In conclusion …

• There are questions that cannot be answered without “other” evidence – Concerning the embodied and embedded ways in which

persons understand and experience illness and its management, and how services and systems do and could condition these realities

• Such evidence can be generated through research (e.g., qualitative research) and/or expert opinion – Which can be more or less relevant – And more or less robust

• The quality of such evidence is a function – Of acts and actors – And of evidence systems

• If we need this evidence, we need systems to support its quality and use 15

References

• Upshur, RE. (2001). The status of qualitative research as evidence. Morse, Janice M., Janice Swanson, and Anton J. Kuzel. (Eds.) The nature of qualitative evidence. Sage, 2001: 5-26

• Madjar, I & Walton, JA. (2001). What is problematic about evidence? Morse, Janice M., Janice Swanson, and Anton J. Kuzel. (Eds.) The nature of qualitative evidence. Sage, 2001: 5-26

• Chandler, JK., Davidson, AI., Harootunian, HD. (1994) Editor’s Introduction, Questions of evidence: Proof, practice, and persuasion across the disciplines. Chicago: University of Chicago Press, 1-8

• The Joanna Briggs Institute. (2014) Joanna Briggs Institute Reviewers’ Manual: 2014 edition. The Joanna Briggs Institute

16

References

• Sandelowski, M., Voils, C. I., Leeman, J., & Crandell, J. L. (2012). Mapping the mixed methods–mixed research synthesis terrain. Journal of mixed methods research, 6(4), 317-331.

• Pang, T., Sadana, R., Hanney, S., Bhutta, Z. A., Hyder, A. A., & Simon, J. (2003). Knowledge for better health: a conceptual framework and foundation for health research systems. Bulletin of the World Health Organization, 81(11), 815-820.

• Dobrow, MJ., Miller, FA., Frank, C., Brown, AD., (2015) Understanding Relevance of Health Research: Considerations in the Context of Research Impact Assessment, Commissioned by the Ontario SPOR Support Unit, IHPME White Paper Series

17

Questions?

18