A Researcher’s Perspective on Quality Improvement Research

Neil R. Powe, MD, MPH, MBAProfessor of Medicine, Epidemiology and Health

Policy and ManagementDirector, Welch Center for Prevention, Epidemiology and Clinical Research

Johns Hopkins School of Medicine and Bloomberg School of Public Health

Outline• Research and quality improvement

• How did we get ourselves into this conundrum?

• It is not new and it is not just a problem for healthcare

• Are all QI efforts created equal?

• What is the evidence that quality improvement research does harm?

• Options and Solutions

Definitions• Research: A systematic investigation including

research development, testing and evaluation designed to develop or contribute to generalizable knowledge (45 CFR 46.102)

• Quality improvement: a systematic pattern of actions that is constantly optimizing productivity, communication and value within an organization in order to achieve the aim of measuring the attributes, properties, and characteristics of a product/service in the context of the expectations and needs of customers and user of that product. (Institute of Medicine)

Research versus Quality ImprovementResearch Quality Improvement

Goal Advance general knowledge in academic, scientific or professional community: generate, evaluate or confirm explanatory theory or conclusion and invite critical appraisal by peers through presentation/debate in public forums

Improve patient care, a clinical program or service: identify specific services, protocols, clinical practices, or clinical processes/outcomes within a department, clinical program or facility for improvement

Lit review

Organized review of relevant literature Available literature and comparative data, or clinical programs, practices or protocol at other institutions to design improvement plan; do not plan full scientific literature reviews

Research design

Leads to scientifically valid findings (control groups, random subject selection, statistical tests)

Established quality improvement methods (e.g. PDSA cycle) aimed at producing change. Does not include sufficient research design elements to support scientifically valid findings

Benefit Most patient/subjects not expected to derive personal benefit from the knowledge gained

Most patients who participate are expected to benefit from the knowledge gained

Risk May impose risk or burden on patients Does not impose any risk or burden on patients

Modified from Cambridge Health Alliance

The Overlap is the Issue

Research

QualityImprovement

How did we got ourselves into this conundrum?

Statement on Health Care Research and Quality ImprovementJohn Eisenberg, M.D., Administrator, AHCPR. President's Advisory Commission on Consumer Protection and Quality in the Health Care Industry, Quality Improvement Environment Subcommittee, Nov 18, 1997, Washington, D.C.

“The improvement in health care quality will depend not only on traditional methods like excellence in education, continuing education, professional standards and principled practice behavior, as important as they are. It will also depend on our developing the science of clinical practice, on our taking a fresh look at ways of measuring and improving care, and on our holding both the practice of medicine and the practice of quality improvement to the same levels of scrutiny and evidence as we hold technical clinical innovations.”

Call for Science of QI

Rigor in QI MethodsEccles M, Grimshaw J, Campbell M, Ramsay C. Research designs for studies evaluating the effectiveness of change and improvement strategies Qual Saf Health Care 2003;12:47-52 “The methods of evaluating change and improvement strategies are not well described. The general principle under-lying the choice of evaluative design is, however, simple—those conducting such evaluations should use the most robust design possible to minimize bias and maximize generalizability.”Institute of Medicine. The Learning Healthcare System: Workshop Summary. Washington, DC: 2007 A roundtable compendium encouraging movement toward increasingly sophisticated research designs other than RCTs to assess quality improvement or patient-safety initiatives. Speroff T, O’Connor GT. Study designs for PDSA quality improvement research. Qual Manag Health Care.2004;13:17-32.“Improving the rigor of the quality improvement literature will build a stronger foundation and more convincing justification for the study and practice of quality improvement in health care.”

Guidelines for QI Appraisal

Speroff T, James BC, Nelson EC, Headrick LA, Brommels. Guidelines for Appraisal and Publication of PDSA Quality Improvement. Quality Management in Health Care. 2004. 13(1):33-39.

“Users of quality improvement could benefit with markers to gauge the "best" science. …. Analogous to existing research guidelines, the PDSA quality improvement guidelines will provide researchers and reviewers with succinct standards of methodological rigor to assist in critical appraisal of quality improvement protocols and publications.”

Books on the Science on QI:

Grol R, Baker R, Moss R. editors. Quality Improvement Research: Understanding the science of change in health care. Blackwell BMJ Books

It is not new and it is not just a problem for healthcare

Action Research• Kurt Lewin then a professor at MIT, first coined

the term “action research” in 1944 • “Action Research and Minority Problems” 1946.

– Paper described “a comparative research on the conditions and effects of various forms of social action and research leading to social action” that uses “a spiral of steps, each of which is composed of a circle of planning, action, and fact-finding about the result of the action”.

Action Research & QI Similarities Quality improvement

• Team effort of identifying opportunities for improvements, measuring performance, and involving the frontline providers and staff members to find ways to improve performance. • Goes through repetitive cycles of measuring performance, testing change concepts and then re-evaluation of outcomes measures

Action research•Reflective process of progressive problem solving led by individuals working with others in teams or as part of a "community of practice" to improve the way they address issues and solve problems. •Can be undertaken by larger organizations or institutions, assisted or guided by professional researchers, with the aim of improving their strategies, practices, and knowledge of the environments within which they practice.

Action Research Operates Occurs at Many Levels

• 1st person research: my research on my own action aimed primarily at personal change

• 2nd person research: our research on our group (family/team) aimed primarily at improving the group

• 3rd person research: scholarly’ research aimed primarily at theoretical generalization and/or large scale change

Action Research QI

Action Research Goes on in Many Places

• Medicine– Quality improvement

• Public Health– Public health practice– Community-based participatory research

• Education – Primary, secondary and higher education

All QI efforts are not created equal • Tension exists between those that

are more driven by the researcher’s agenda to those more driven by participants

• For QI researcher, the Study in PDSA is paramount

• For the QI doer, the Do and Act in PDSA are paramount

What is the Evidence That Quality Improvement Research Does Harm? • We do not really know!

• Possible threats– Pressure of cost containment– Misapplication of an intervention to different

patient populations– Unintended consequences

• How much risk is involved?

The Risk Escalation Principle of Protection

As the risk of the research rises about the threshold of “minimal risk” there are restrictions and additional protections, particularly for vulnerable subjects

Level of Risk Determines Level of Review by Institutional Review Boards

Minimal Risk

Full IRB Committee Review

Expedited Review

Exempt from Review

Definition of Minimal Risk in Subpart A of Code of Federal Regulations

• Minimal risk means that the probability and magnitude of harm or discomfort anticipated in the research are not greater in and of themselves than those ordinarily encountered in daily life* or during the performance of routine physical or psychological examinations or tests

45 CFR 46.1029(i)

*Daily lives of the subjects of the research, not healthy individuals

Which is greater?

• The probability and magnitude of harm or discomfort anticipated in the research

• Harm or discomfort ordinarily encountered in daily life or during the performance of routine physical or psychological examinations or tests

OR

Why is this a problem? One example

• Practice Improvement Modules (PIMs) are required for board certification by American Board of Internal Medicine and others for Maintenance of Certification

• PIMs provide physicians with a way to apply quality measurement to their practices, and then use the resulting data to take actions to improve care.– Physicians who are competent in practice-based

learning and improvement and systems-based practice will be able to demonstrate the quality of care they deliver with data from their practice, and use that data for improvement, reporting, and upholding our professional value of accountability.

Why is this a problem? One example • PIMs are Web-based, self-evaluation tools that guide

physicians through collecting data from their own practice, using chart reviews, patient surveys and a practice system survey to create a multi-dimensional practice performance assessment. – ABIM uses the data submitted electronically by the

physician to provide an interactive report. – Detailed performance data serve as the basis for

selecting areas for improvement and developing an improvement plan.

– After implementing and testing the improvement plan, physicians report the impact of the implemented changes to ABIM.

• What would be the impact on IRB review on doctors practicing in different settings?

Policy Options• Establish separate committees to review QI

projects

• IRB guidelines and submission of all projects

• Hang a sign on front door of every hospital and physician’s office in the country

Superior Health Care Front Door Sign

Thank you for deciding to receive your care here. Your safety and health is our primary concern. We strive to improve the care that you will receive here by engaging in a rigorous process of quality improvement in which we continuously plan new initiatives, implement them, study them rigorously, and act on what we find that works. By receiving your care here, you become a fully engaged participant in this process.