Examination of the Interrater Reliability of a Palliative Care Assessment Tool in Patients at Hospital Admission:
A Pilot Study
April 28 2014State of the Art Nursing Conference
Angie Andersen DNP, ACNP-BC
Angela Andersen has no financial interest or arrangement that would be considered a conflict of interest.
Faculty Disclosure
Research Team Angela Andersen, DNP, ACNP-BC, Nurse Practitioner, Palliative Care
Department, The Nebraska Medical Center, Omaha, NE.
Mary Parsons, PhD, RN, Associate Professor and Chair DNP Program, Creighton University School of Nursing, Omaha, NE.
Regina Nailon PhD, RN, Clinical Nurse Researcher, The Nebraska Medical Center Omaha, NE.
Sue Ann Gaster BSN, RN, Staff Nurse, Adult Progressive Care Unit, The Nebraska Medical Center, Omaha, NE.
Rachael Mooberry BSN, RN, Staff Nurse, Oncology-Hematology Specialty Care Unit, The Nebraska Medical Center, Omaha, NE.
Jane Meza, PhD, Professor College of Public Health Biostatistics, University of Nebraska Medical Center, Omaha, NE.
Goal of Palliative Care
Improve quality of life for patients and their families facing the problems associated with serious or life-threatening illness, through the prevention and relief of sufferingWorld Health Organization ,
2011
Interdisciplinary Team
Palliative care teams utilize an interdisciplinary approach in which physicians, nurses, chaplains, social workers, and other allied health professionals provide care
Weissman & Meier, 2011
Palliative Care Consultation
Treat pain and other symptoms that can cause complications in hospitalized patients
Establish goals for careSupport family members in crisisPlan for safe transitions from
hospital to other settingsWeissman & Meier, 2011
Levels of Care
Three levels of palliative care: Primary▪ Basic skills and competencies required to manage the day-
to-day patient care Secondary ▪ Treating physician refers to a specialist-level palliative care
provider for management of complex or difficult problems Tertiary▪ Education and research
Von Gunten & Lupu, 2004; Weissman & Meier, 2011
How We Die Nearly half of all Americans die in a hospital 7 out of 10 Americans say they would prefer
to die at home Only 25 % of Americans actually die at
home More than 80% of patients with chronic
disease say they want to avoid hospitalization and intensive care when they are dying Centers for Disease Control, 2005; Dartmouth Atlas of Health Care, 2005
How We Die Researchers examined medical records for
840,000 people 66 or older who died in 2000, 2005, and 2009: Increase use of hospice program in 2009, but more
than a quarter of hospice use was for 3 days or less, and
40% of those late referrals followed a hospitalization with an intensive-care stay
Patients receive aggressive care until time of death and did not receive full benefit of hospice care or program
Teno et al., 2013
Literature Review
Extensive body of evidence demonstrates difficulties in providing adequate pain and symptom management, as well as inconsistent communication for hospitalized patients with serious or life threatening conditionsTilden et al., 1995; Hanson et al., 1997; Claessens et al., 2000; Lynn et al., 2000; Norton & Talerico, 2000; Norton et al., 2002
Literature Review
The aggressive care provided in hospitals during the last year of life accounts for approximately 12% of the U.S. health care budget and 27% of Medicare expenditures Centers Disease Control, 2009
Left unchecked, it is projected that health care spending will increase 25% by 2030, largely because of the aging population and chronic disease Centers Disease Control, 2009
Literature Review
Evidence suggests patients and families with serious or life-threatening illness who received palliative care interventions along with standard care reported: Improved physical and psychological symptoms Improved quality of life Longer median survival time Improved family caregiver well-being
Lautrette,2007 ; Wright et al., 2008; Bakitas et al., 2009; & Temel et al., 2010
Early Identification of Needs
Despite the evidence, transition from disease-directed treatment to an emphasis on palliative care often occurs within days of end of life, if at all
Hui et al., 2010; Reville et al., 2010; Hi et al., 2012
Early identification of palliative care needs is critical for clinicians to provide appropriate and timely interventions directed at the specific level of palliative care required by the patient and their family Weissman & Meier, 2011
Assessment Instrument
General lack of assessment techniques that would equip providers to identify palliative care needs in hospitalized patients
Although a variety of palliative care assessment instruments have been developed, non have examined reliability or validity to dateBradley & Brasel, 2009; Fins, Miller et al., 1999; Fins, Schwager et al., 2000; Imhof, Kaskie, & Wyatt, 2007
Evidence-based Instruments
Investigating the psychometric properties of an instrument is a common standard prior to implementing the tool in clinical practicehttp://www.jointcommission.org/accreditation/hospitals.aspx
The Centers for Medicare & Medicaid Services and Joint commission on Accreditation of Healthcare Organizations support the use of evidence-based instruments and practices by clinicians caring for hospitalized patientshttp://www.jointcommission.org/accreditation/hospitals.aspx
CAPC Criteria
The Center to Advance Palliative Care (CAPC) developed a set of criteria to identify patients at hospital admission that would be appropriate to receive further palliative care assessment and interventionsWeissman & Meier, 2011
CAPC Criteria
No study has examined the reliability and other psychometric properties of the CAPC criteria
Reliability is a prerequisite for any kind of validity, and is the degree to which measurement error is absent from data Polit & Beck, 2008
Interrater Reliability
Interrater reliability is a specific type of reliability referring to the amount of agreement between different raters. In the case of the CAPC criteria, interrater
reliability is useful to measure whether two (or more) raters independently come to an exact or nearly exact agreement when scoring a patient
Polit & Beck, 2008
Interrater Reliability Although exact agreement of independent raters
is ideal, a small difference in rating is of minor clinical relevancePolit & Beck, 2012
Conversely, if the difference in assessment and scoring between raters increases, it is very likely this will have implications for clinical practicePolit & Beck, 2012
For example, whereas one nurse considers a patient at risk for having unmet palliative care needs and provides intervention to address the specific level of palliative care required by patient and their family, another nurse may regard the patient not at risk and will not consider any further interventions
Purpose
The purposes of this pilot study were twofold: To establish the interrater reliability of CAPC
criteria that identify and trigger primary palliative care assessment at hospital admission in adult patients admitted to oncology and progressive care units who received palliative care services during their hospitalization; and
To describe the CAPC criteria identified most frequently in study patients who met CAPC criteria at hospital admission
Methods Design
Retrospective, descriptive, exploratory Setting and Sample▪ Midwestern academic medical center▪ Randomized, purposive sample▪ Using a sample of patients known to have received palliative care
services strengthened the study design and enhanced the study team’s ability to determine the interrater reliability of the CAPC criteria
▪ A sample size of 100 was adequate to determine the instrument’s reliability using the kappa statistic, along with a 95% confidence interval.
▪ Inclusion Criteria ▪ 19 years or older ▪ Admitted to the adult oncology or progressive care units between January
1 and December 31, 2011 who received palliative care services during their hospitalization
Instrument
The CAPC criteria comprise an instrument for use at hospital admission to identify patients whose conditions warrant a primary palliative care assessment
National consensus panel developed criteria from research findings, national standards, and expert opinionWeissman & Meier, 2011
Instrument
The CAPC criteria has primary and secondary criteria to facilitate identification of patients appropriate for primary palliative care assessment
The starting point for assessing any given patient using the primary and secondary criteria is the identification of patients with potentially life-limiting or life-threatening conditionsWeissman & Meier, 2011
Instrument
Primary Criteria Includes the 5 most important indicators
identified by the consensus panel These criteria are global indicators that
represent the minimum that nurses and clinicians should use to screen patients at risk for unmet palliative care needs at hospital admission
Weissman & Meier, 2011
Primary CriteriaPotentially life-limiting or life-threatening condition
Surprise Question: You would not be surprised if the patient died within 12 months.
Frequent admissions (more than one admission for same condition within 3 months).
Admission prompted by difficult-to-control physical or psychological symptoms (e.g., moderate-to-severe symptom intensity for more than 24 hours).Complex care requirements:• Functional dependency• Complex home support for ventilator• Complex home support for antibiotics• Complex home support feedings
In last 3 months, decline in:• Function (mobility or mental capacity)• Feeding intolerance (nausea, vomiting, or bloating)• Unintended decline in weight (e.g., failure-to-thrive)
Instrument
Secondary Criteria More specific indicators of higher
likelihood of unmet palliative care needs and are designed to be used as supplemental criteria in hospitals with more comprehensive palliative care services available
Secondary Criteria
Admission from long-term care facility or medical foster home
Cognitively impaired elderly (> 70 years) patient with acute hip fracture
Metastatic or locally advanced incurable cancer
Chronic home oxygen use
Out-of-hospital cardiac arrest
Current or past hospice program enrollee
Limited social support
No history of completing an advanced care planning discussion
Procedure
Nurse investigators independently reviewed the medical record of each patient for evidence of CAPC criteria present within 48 hours of patients hospital admission The nurse investigators determined the presence
of one or more CAPC criteria that identified the need for primary palliative care assessment
The principal investigator reviewed all 100 medical records and each co-investigator reviewed 50
Data Analysis
Inter-rater reliability was examined with the kappa statistic, along with a 95% confidence interval
A test for whether kappa is different from zero was also calculated If the p-value < 0.05, we concluded
that the kappa value was significantly different from zero
Oncology Unit Interrater Reliability
Primary Criteria N= 50
YesCount (%)
NoCount (%)
1UTDCount (%)
Kappa Value
p-value
Life-limiting/threatening condition 50 (100%) 0 0 1.000 -
Surprise question2 (4%) 48 (96%) 0 1.000 -
Frequent admissions35 (70%) 15 (30%) 0 0.854 < .0001
Admit difficult-to-control symptoms46 (92%) 4 (8%) 0 0.648 < .0001
Functional dependency22 (44%) 23 (23%) 5 (10%) 0.825 < .0001
Complex home support ventilator0 50 (100%) 0 1.000 -
Complex home support antibiotics0 50 (100%) 0 1.000 -
Complex home support feedings6 (12%) 44 (88%) 0 1.000 < .0001
Last 3 months, decline in function 43 (86%) 5 (20%) 2 (4%) 0.742 < .0001
Last 3 months, feeding intolerance28 (56%) 18 (36%) 4 (8%) 0.817 < .0001
Last 3 months, decline in weight16 (23%) 15 (30%) 19 (38%) 0.757 < .0001
1Unable To Determine
Kappa Value /Agreement:1.000 = perfect 0.99 -0.81 = almost perfect 0.80-0.61 = substantial 0.60 or less = poor
Oncology Unit Interrater Reliability
Secondary Criteria N= 50
YesCount (%)
NoCount (%)
1UTDCount (%)
Kappa Value
p-value
Admission long-term care facility 7 (14%) 43 (86%) 0 0.912 < .0001
Cognitively impaired elderly hip fx. 0 50 (100%) 0 1.000 -
Metastatic or incurable cancer 50 (100%) 0 0 1.000 -
Chronic home oxygen 8 (16%) 42 (84%) 0 0.702 < .0001
Out-of-hospital cardiac arrest 0 50 (100%) 0 1.000 -
Hospice program 0 50 (100%) 0 1.000 -
Limited social support 12 (24%) 38 (76%) 0 0.390 < .0025
No history advance care planning 15 (30%) 70 (53%) 0 0.595 < .0001
1Unable To Determine
Kappa Value /Agreement:1.000 = perfect 0.99 -0.81 = almost perfect 0.80-0.61 = substantial 0.60 or less = poor
Progressive Care Unit Interrater Reliability
Primary Criteria N= 37
YesCount (%)
NoCount (%)
1UTDCount (%)
Kappa Value
p-value
Life-limiting/threatening condition 37 (100%) 0 0 1.000 -
Surprise question 0 36 (97%) 1 (3%) 1.000 -
Frequent admissions 9 (24%) 25 (68%) 3 (8%) 0.703 < .0001
Admit difficult-to-control symptoms 36 (97%) 1 (3%) 0 1.000 -
Functional dependency 24 (65%) 10 (27%) 3 (8%) 0.311 0.0067
Complex home support for ventilator 0 50 (100%) 0 1.000 -
Complex home support antibiotics 0 50 (100%) 0 1.000 -
Complex home support feedings 3 (8%) 33 (89%) 1 (3%) 0.844 < .0001
Last 3 months, decline in function 24 (65%) 8 (22%) 5 (13%) 0.392 < .0001
Last 3 months, feeding intolerance 13 (35%) 21 (57%) 3 (8%) 0.712 < .0001
Last 3 months, decline in weight 9 (24%) 21 (57%) 7 (19%) 0.479 < .0001
Kappa Value /Agreement:1.000 = perfect 0.99 -0.81 = almost perfect 0.80-0.61 = substantial 0.60 or less = poor
1 Unable To Determine
Progressive Care Unit Interrater Reliability
Secondary Criteria N= 37
YesCount (%)
NoCount (%)
1UTDCount (%)
Kappa Value
p-value
Admission long-term care facility 9 (24%) 28 (76%) 0 0.924 < .0001
Cognitively impaired elderly hip fx. 1 (3%) 36 (97%) 0 1.000 < .0001
Metastatic or incurable cancer 10 (28%) 26 (72%) 0 0.933 < .0001
Chronic home oxygen 9 (24%) 28 (76%) 0 0.853 < .0001
Out-of-hospital cardiac arrest 1 (3%) 36 (97%) 0 0.654 < .0001
Hospice program 0 37 (100%) 0 1.000 -
Limited social support 10 (27%) 17 (46%) 10 (27%) 0.510 < .0001
No history advance care planning 19 (51%) 18 (49%) 0 0.837 < .0001
Kappa Value /Agreement:1.000 = perfect 0.99 -0.81 = almost perfect 0.80-0.61 = substantial 0.60 or less = poor
1Unable To Determine
Results
Study sample revealed most frequently identified CAPC Criteria:▪Life-limiting condition▪Surprise question
Discussion
Nurse investigators had perfect to substantial agreement for the majority of the CAPC criteria
Perfect to substantial agreement provides confidence in nurses’ abilities to administer and score the CAPC instrument for the study population
Discussion
Prior to making inferences about interrater reliability of CAPC criteria, it is important to note the limitations of the CAPC instrument Poor level of agreement for four criteria:▪ Limited social support▪ Functional dependency▪ In the last 3 months decline in function▪ In the last 3 months decline in weight
Lack of operational definitions
Discussion
In clinical practice, it is common that a team of interdisciplinary clinicians provide care to patients and their families Nurses on the team have an essential role in
identifying unmet needs of patients and coordinating services
Therefore it is essential that any instrument used to assess patients for palliative care needs has findings that are repeatable between nurses
Study Limitations
Retrospective study design may have contributed to the nurse investigators’ inability to determine the presence or absence of each criterion Quality of documentation Investigators knowledge and experience
Generalizability of study findings: Patients who received palliative care Definition of “hospital admission”
Clinical Implications Identification of palliative care needs is
necessary for nurses and other clinicians to be able to provide interventions directed at the specific level of palliative care required by the patient and their family
Establishing interrater reliability of the CAPC criteria is a necessary first step in determining the utility of having registered nurses conduct the screening at hospital admission
Research Implications Beginning of a process that will contribute to the
availability of data that describe the characteristics of hospitalized patients appropriate for further palliative care assessment and intervention
Future examination of the interrater reliability of CAPC criteria: Other patient populations Formal education of nurse investigators to increase
understanding of what each criterion is intended to measure
Describe clinically relevant differences between nurses’ disagreements
Conclusions
Nurses play an essential role in identifying hospitalized patients who are at risk for having unmet palliative care needs
Establishing interrater reliability of the CAPC criteria is essential to determining the utility of having RN’s conduct the screening at hospital admission of patients who are at risk for unmet palliative care needs
Conclusions
Based on the study findings, it is realistic and suitable for nurses to administer and score the CAPC criteria at hospital admission
Implementation of an established instrument will provide the structure and process needed to ensure consistent and timely identification of patients at risk for having unmet palliative care needs
Conclusions
This study is a first attempt at establishing psychometric properties of the CAPC criteria to identify and trigger further palliative care assessment at hospital admission
Questions?
