J CIim R#&abl Vol. 46, No. 4, pp. 379-393, 1993 Rintcd in Great Britain. Ail rights reserved 0895-4356/93 $6.00 + 0.00 Copyright 0 1993 Pcrgamon Press Ltd THE DUKE SEVERITY OF ILLNESS (DUSOI) FOR MEASUREMENT OF AND COMORBIDITY CHECKLIST SEVERITY GEORGE R. PARKERSON JR,* W. EUGENE BROADHEAD and CIIIU-KIT J. TSE Department of Community and Family Medicine, Box 2914, Duke University Medical Center, Durham, NC 27710, U.S.A. (Received in revised fom 30 September 1992) AI&rat--The Duke Severity of Illness Checklist (DUSOI) was evaluated on 414 primary care adult patients using data collected both by medical providers at the time of the patient visit and later by a chart auditor. Severity scores for individual diagnoses were determined by summing the ratings for four non-disease-specific parameters: symptom level, complications, prognosis without treatment, and expected response to treatment. Mean diagnosis severity scores (scale O-100) among the 21 most prevalent diagnoses varied from a low of 13.9 for menopausal syndrome to a high of 43.0 for sprains and strains. An overall severity score was calculated by combining diagnosis severity scores and giving highest weights to the most severe diagnoses. Provider- generated overall severity scores (mean = 43.3) and auditor-generated overall severity scores (mean = 38.9) were significantly correlated (coefficient of agreement = 0.59, p < 0.0001). Diagnoses varied in their individual contribution to the overall severity score, from 8.9% for lipid disorder to 90.0% for sprains and strains. Separate comorbidity severity scores were calculated to measure the severity of all of each patient’s health problems except the diagnosis under study. For example, patients with menopausal syndrome had co-existing health problems which generated a high mean comorbidity severity score of 43.2, while patients with sprains and strains had a low mean comorbidity score of 4.7. The DUSOI Checklist can be used in the clinical setting by both providers and auditors to produce quantitative severity scores (by diagnosis, overall, and for comorbidity) which are based entirely upon clinical judgment. This method should be useful in controlling for severity of illness in clinical studies and indicating the outcome of medical care in terms of reduction in severity of illness following medical interventions. Severity of Illness Index Comorbidity INTRODUCTION Measurement of severity of illness and co- morbidity has become increasingly important during the past 20 years, as the quality and cost of health care have become prime issues for the public and third party payers. Most of the original methodologies for measurement of these factors were developed for use in the inpatient hospital setting with data abstracted *Author for correspondence. Health status from medical records. Kaplan and Feinstein classified clinical ailments by three grades of severity in their study of the effect of comorbid- ity on diabetic outcomes [l]. Gonnella et al. advanced the staging concept, which defines different levels of severity for specific diseases, based primarily upon manifestations and com- plications of each disease [2-4]. Horn et al. produced the Severity of Illness Index, a disease- generic method which determines severity from seven indicators: stage of the principal diagnosis, its interactions with other diagnoses, 379
Transcript
J CIim R#&abl Vol. 46, No. 4, pp. 379-393, 1993 Rintcd in Great Britain. Ail rights reserved
THE DUKE SEVERITY OF ILLNESS (DUSOI) FOR MEASUREMENT OF
AND COMORBIDITY
CHECKLIST SEVERITY
GEORGE R. PARKERSON JR,* W. EUGENE BROADHEAD and CIIIU-KIT J. TSE Department of Community and Family Medicine, Box 2914, Duke University Medical Center,
Durham, NC 27710, U.S.A.
(Received in revised fom 30 September 1992)
AI&rat--The Duke Severity of Illness Checklist (DUSOI) was evaluated on 414 primary care adult patients using data collected both by medical providers at the time of the patient visit and later by a chart auditor. Severity scores for individual diagnoses were determined by summing the ratings for four non-disease-specific parameters: symptom level, complications, prognosis without treatment, and expected response to treatment. Mean diagnosis severity scores (scale O-100) among the 21 most prevalent diagnoses varied from a low of 13.9 for menopausal syndrome to a high of 43.0 for sprains and strains. An overall severity score was calculated by combining diagnosis severity scores and giving highest weights to the most severe diagnoses. Provider- generated overall severity scores (mean = 43.3) and auditor-generated overall severity scores (mean = 38.9) were significantly correlated (coefficient of agreement = 0.59, p < 0.0001). Diagnoses varied in their individual contribution to the overall severity score, from 8.9% for lipid disorder to 90.0% for sprains and strains. Separate comorbidity severity scores were calculated to measure the severity of all of each patient’s health problems except the diagnosis under study. For example, patients with menopausal syndrome had co-existing health problems which generated a high mean comorbidity severity score of 43.2, while patients with sprains and strains had a low mean comorbidity score of 4.7. The DUSOI Checklist can be used in the clinical setting by both providers and auditors to produce quantitative severity scores (by diagnosis, overall, and for comorbidity) which are based entirely upon clinical judgment. This method should be useful in controlling for severity of illness in clinical studies and indicating the outcome of medical care in terms of reduction in severity of illness following medical interventions.
Severity of Illness Index Comorbidity
INTRODUCTION
Measurement of severity of illness and co- morbidity has become increasingly important during the past 20 years, as the quality and cost of health care have become prime issues for the public and third party payers. Most of the original methodologies for measurement of these factors were developed for use in the inpatient hospital setting with data abstracted
*Author for correspondence.
Health status
from medical records. Kaplan and Feinstein classified clinical ailments by three grades of severity in their study of the effect of comorbid- ity on diabetic outcomes [l]. Gonnella et al. advanced the staging concept, which defines different levels of severity for specific diseases, based primarily upon manifestations and com- plications of each disease [2-4]. Horn et al. produced the Severity of Illness Index, a disease- generic method which determines severity from seven indicators: stage of the principal diagnosis, its interactions with other diagnoses,
379
380 GEORGE R. PARKERKIN JR et al.
patient rate of response to treatment, residual impairment after therapy, complications, dependency of the patient upon medical care, and extent of non-operating room procedures required [5,6].
More recently, Horn’s group has developed the Computerized Severity Index, a disease- specific tool which uses objective clinical find- ings to rate severity for each ICD-9-CM classification code [7]. Computerized scoring provides a severity level score for each disease and a combined score for overall severity from all comorbid conditions [7].
Charlson et al. emphasized the importance of physician clinical judgment in the determi- nation of severity [8,9]. Judgments by resident physicians were used to assess severity based upon how sick they considered each patient. Residents also rated patient functional ability and gave their prognosis for the patient’s j-year survival. Follow-up studies showed that predic- tors of l-year survival after hospitalization were functional ability, severity of illness, extent of comorbid disease and physician prognosis for survival [9]. The importance of comorbidity also was demonstrated by Greenfield et al., in their studies of the Comorbidity Index, which measures baseline comorbid severity, acute exa- cerbations, and patient functional status [lo].
In the primary care setting, Barsky et al. studied severity by auditing ambulatory medical records to rate the severity of each diagnosis according to the amount of disease, prognostic threat to life, number of organs involved, dis- ability, complications, and seriousness of treat- ment [1 11. Parkerson et al. developed the Duke Severity of Illness Scale (DUSOI) as a chart audit system in which severity is based upon estimates of treatability and prognosis, compli- cations, and symptom level for each diagnosis, and in which the diagnosis severity scores are combined into an overall severity score for each patient [12].
Current interest in severity has been focused primarily upon the development of ambulatory case-mix measures. Examples are the Ambulat- ory Visit Groups (AVGs) by Fetter et al. [13], the Ambulatory Severity Index (ASI) by Horn [14], the Products of Ambulatory Care (PACs) by Tenan et al. [15], the Ambulatory Patient Groups (APGs) by Averill et al. [16], and the Ambulatory Care Groups (ACGs) by Starfield et al. [17].
These are classification systems which are based upon different combinations of factors,
such as patient age and gender, diagnoses, medi- cal or surgical procedures, medications, and types of encounter. They are designed to quan- tify patient population levels of severity which can be used for health services research, improvement of ambulatory care management, and determination of appropriate reimburse- ment for medical service [18] . None of these measures incorporates a direct assessment of severity based upon the clinical judgment of the patient’s medical provider.
Most of the current methods depend upon encounter form information and/or medical record audits for their data. While these methods have the advantage of relative ease of use, without the necessity of burdening the medical provider with data collection, they have the distinct disadvantage of producing second- hand information from records which are often a very abbreviated version of what the medical provider knows about the patient.
The present study was undertaken to develop a severity of illness measure which would be completed by the medical provider at the time of the patient visit and which would produce a quantitative assessment of how sick the patient is, based upon the knowledge and judgment of the provider. A new checklist version of the DUSOI was evaluated by comparing severity scores generated by medical providers with severity scores from medical record audit.
METHODS
The study population consisted of ambu- latory adult patients visiting the Caswell Family Medical Center, a rural primary care community health center in the small town of Yanceyville, North Carolina. The centre provides health care for approximately 3000 patients of all ages, about half of whom are female, and whose racial mix is about half black and half white. Patients present with a full range of health problems, but obstetrical care is not provided.
Data were collected during the 8-month period ending in April 1991. Consenting patients aged 1865 years who presented to the clinic for their usual health care, and who demonstrated adequate literacy by com- pleting a 17-item demographic questionnaire, were included in the study. Literacy was im- portant because participants were required to complete a multi-item questionnaire for a com- ponent of the study which is reported elsewhere
Duke Severity of Illness Checklist 381
[ 191. A convenience sample was chosen in an attempt to provide at least 30 patients in each of 12 categories, which combined gender, black and white race, and three age groups (18-33, 3449, and 50-45 years).
During the study period medical care was provided by two family physicians, two general internists, and one physician assistant. All of these providers participated in the study by completing the DUSOI checklist and DUSOI analog scale (both described below) just after seeing patients who had agreed to participate. The first author of this report, a family phys- ician not associated with the clinic, served as the principal auditor. He audited all of the medical records using the same DUSOI checklist that was used by the providers. Two other non-clinic family physicians audited the records for data which were used only in the intra- and inter- auditor analyses. Audits were performed from photocopies of medical records which had patient identification removed, except for ID numbers. Portions for audit included the problem list, medication list, laboratory reports, and progress notes for the visit at which the patient entered the study and for the preceding year.
The DUSOI is an instrument for measuring illness severity and comorbidity in ambulatory patients. The DUSOI score reflects each patient’s “burden of illness” on the day of the patient visit and during the preceding week, i.e. at a given point in time, for all recorded health problems. Initially the DUSOI was described as a chart audit modality [12]. Since then it has been revised into a checklist format that can be used both by clinicians at the time of the patient encounter, and by medical record auditors retrospectively (see Appendix A). The non- disease-specific parameters for judging severity have been refined since the initial report by separating treatability and prognosis into separ- ate components, so that the current parameters include symptom level, complications, progno- sis without treatment, and treatability.
The symptom level parameter indicates the symptomatic state of the patient on the day of the severity rating and during the preceding week. The complications parameter shows the level of complications present during that same period of time, with a complication defined as a health problem which is secondary to another health problem, but which does not warrant listing as a separate problem in the rater’s judgment. Prognosis indicates the expected out-
come for the patient in terms of disability or threat to life during the 6 months following the rating encounter, if the health problem were to go untreated. Prognosis without treatment is considered to be an important indicator of the basic seriousness of the health problem, even when treatment is available and will be given. The fourth severity parameter, treatability, indi- cates the prognosis with treatment in terms of the provider’s perceived need for treatment and the expected response to treatment if it is needed. High severity parameter ratings indicate the presence of more symptoms, more compli- cations, worse prognosis without treatment, and worse expected response to treatment.
Although the DUSOI severity parameters are themselves non-disease-specific, severity ratings are made separately for each of the patient’s health problems. The first step in completing the checklist is to record each diagnosis or health problem. The order in which the problems are entered .on the form is not important, and any number can be included. All diagnoses or health problems are listed which are active at the time of the visit or during the preceding week, includ- ing those chronic conditions which were not managed on the day of the visit. For example, a patient with pneumonia might have the chronic problems hypertension and obesity, which were not mentioned in the progress note on the day of the index visit, but which were listed on the problem list and/or elsewhere in the progress notes prior to the index visit. The patient’s medical provider may be able to recall this information, but the auditor will have to obtain it by reviewing the problem list, medi- cation record, and progress notes for the preced- ing year.
Health problems are listed on the DUSOI checklist in the words of the provider. They may include standard diagnostic terms, symptoms, and/or ill-defined health problems. No standard terminology is required and no explicit diagnos- tic criteria are necessary. Definition and labeling of health problems is based upon the implicit judgment of the clinician who sees the patient. When the chart audit approach is used, the auditor accepts the diagnostic labeling made in the record by the provider.
For the purpose of data description and analysis, any standard diagnostic classification system can be utilized. In the present study, the health problems identified by the providers were coded by a data technician using the International Classification of Health Problems
382 GEORGE R. PARKERSON JR et al.
in Primary Care (ICHPPC) [20]. This system provides 371 rubies for the problems seen most commonly in the primary care setting [20].
After listing all health problems, the next step in completing the DUSOI checklist is to check the one of the five response options which best represents the patient’s clinical status, for each of the four severity parameters for each health problem listed. There are no specific criteria which can be used by the auditor or provider for estimating the levels of severity for each par- ameter. Ratings are based entirely upon the implicit clinical judgment of the rater.
Although manual scoring is possible, comput- erized scoring was used in the present study. The DUSOI diagnosis severity score for each diagnosis is computed by summing the four parameter scores (each of which can range from 0 to 4), dividing by 16, and multiplying by 100 to produce a score on the scale of 0 for lowest to 100 for highest severity.
The DUSOI overall severity score for each patient is computed on a scale of O-100 by using an equation which gives full weight to the diagnosis or health problem with the highest diagnosis severity score, and which gives pro- gressively diminishing weights to diagnoses with lower scores. This method allows entry of an infinite number of diagnoses for each patient, while drastically limiting increments to the over- all severity score by diagnoses which are causing the least burden of illness. The overall severity equation is as follows:
This process generates for each patient a series of diagnosis severity scores to quantitate severity of each of their health problems, and also an overall score which is a quantitative measure of severity resulting from all their illnesses.
A DUSOI comorbidity severity score can be computed by using the same equation as that for overall severity except that the score of the diagnosis of principal interest is omitted from the calculation. For example, when a patient or group of patients is being studied for depression, the diagnosis severity score for depression would be considered separately from the comorbidity severity score, which would be computed for each patient from the scores of all the diagnoses except depression.
The providers in the present study also com- pleted a DUSOI analog scale which consists of a line 100 mm in length with zero for lowest overall severity and 100 for highest overall severity. (See Appendix B.) The provider placed a mark along the line to indicate a rating for the patient’s overall severity of illness during the preceding week, and the score was determined by measuring the distance from the zero end of the line to the mark.
During the study period one subgroup of patients made return visits to the clinic and completed the questionnaire packet at both visits. Data from the initial visit were used in the analyses involving the entire study group, and data from both visits were used in the test-retest analyses.
Intraclass correlation coefficients (ICC) de- rived from a two-way mixed effects analysis of variance model assuming fixed raters’ effects [21] were used to measure intra- and inter-rater reliability on the chart-audited data and also to measure the agreement between audited and provider-generated DUSOI scores. Excellent agreement is indicated by coefficients greater than 0.75; fair agreement, by coefficients between 0.40 and 0.75; and poor agreement, by coefficients less than 0.40 [22].
Spearman rank-order correlation coefficients were used to demonstrate the associations between DUSOI scores and sociodemographic variables and also between test and retest DUSOI scores. The Student t-test was used to test for differences in continuous variables such as age, and the chi-square was used to test for differences in categorical values.
RESULTS
Study population Of the 561 patients who were asked to partici-
pate in the study, 534 (95.2%) consented, and 414 (73.8%) were included in the analyses. The 120 consenting patients who were excluded from
Duke Severity of Illness Checklist 383
the analyses included 50 who were not s&i- ciently literate to complete the questionnaire, 37 who gave incomplete demographic data, 28 who were too sick to participate, and 5 for whom the providers did not complete the DUSOI checklist. Most patients were evaluated by one of three providers, i.e. 39.6% by one of the family physicians, 26.3% by the physician assistant, and 21.7% by one of the general internists. The remaining 12.4% were seen by the other two physicians.
The 414 study patients had a mean age of 40.5 f 13.1 SD years, with 33.6% aged 18-33 years, 39.1% aged 34-49 years, and 27.3% aged 50-65 years. Women constituted 58.7% of the group; black patients, 47.1%; and white patients, 52.9%. Of the group 56.6% were married; 22.8% never married; 9.5% divorced; 6.6% separated; and 4.6% widowed.
Distribution of patients by educational level showed 32.5% who had less than a high school education, 35.9% who were high school gradu- ates without further training, and 31.6% who had education beyond high school. Fifty-seven percent had full-time jobs, 9.3% had part-time jobs, and 6.6% were looking for jobs; 10.8% were keeping house; 2.9% were going to school; 3.7% were retired; 8.1% were disabled; and 1.7% were in miscellaneous categories.
Data from a subgroup of 54 patients who made return visits to the clinic were used for test-retest analyses. These patients had a mean age of 43.8 f 14.2 years, and were 64.8% womenand 31.5% blacks. They included 51.8% who were married, 72.2% with at least a high school education, and 35.9% who were working full-time.
Prevalence of health problems Providers reported 803 health problems for
the 414 patients, for a mean of 1.94 + 1.15 SD per patient. Only 47.6% of the patients had one problem; 26.8% had two; 14.0% had three; 8.2% had four; 2.4% had five; and 1.0% had six problems. The five diagnoses with the highest prevalence were hypertension (28.0%), diabetes mellitus (9.2%), tobacco abuse (9.2%), lipid disorder (7.7%), and acute bronchitis (7.7%).
The distribution of the 21 most prevalent health problems and the extent of their co- morbid relationships are shown in Table 1. For example, hypertension was diagnosed in 116 patients, but only 17 of these had hyper- tension alone. Twenty-four of the hypertensives
also had diabetes; 8 had tobacco abuse; 16 had a lipid disorder; and the others had a variety of co-existing illnesses. Likewise, only 1 of the 38 diabetic patients had diabetes alone. Twenty-four of the diabetics also had hyper- tension; 2 had tobacco abuse; 7 had a lipid disorder; etc.
Severity of illness scores The DUSOI diagnosis severity of illness
scores for the 21 most prevalent health problems are shown in Table 2, arranged in descending rank-order by level of severity. Mean scores ranged from a high of 43.0 f 11.8 SD for sprains and strains to a low of 13.9 f 10.7 SD for menopausal syndrome. Also severity scores varied widely within each diagnostic category, e.g. ranges of 75.0 for hypertension, 81.2 for diabetes, and 93.8 for alcohol abuse.
Each of the diagnosis DUSOI scores was calculated as the mean of the four severity parameter ratings checked off by the provider. The relative contribution of each of these four parameters to the total diagnosis score is shown in Table 3. Symptom level contributed the most toward severity for acute illnesses, e.g. 60.6% for acute upper respiratory infection, and least for chronic conditions, e.g. 6.7% for hyper- tension. Complications contributed the most for chronic conditions, such as alcohol abuse with 14.8%, and the least for acute illnesses, several of which had a zero contribution, e.g. headache and vaginitis, indicating no compli- cations. Ratings for prognosis without treat- ment (expected disability or death) contributed most toward severity scores for chronic diseases, such as ischemic heart disease (42.2%), and least for acute illness such as vaginitis (2.6%). Treatability (expected poor response to treat- ment) was the most important determinant of severity for chronic conditions, e.g. lipid dis- order (84.5%), and least important for acute conditions, e.g. acute upper respiratory infec- tion (26.1%). Overall, for the 803 health prob- lems, treatability was the most important contributor to diagnosis severity scores (37.7%), and complication level was the least important (5.2%).
DUSOI overall severity scores calculated from weighted diagnosis severity scores are shown in Table 4 for patients with each of the 21 most prevalent health problems. These overall severity scores ranged from a high of 60.7 + 10.9 SD for depression, to a low of 36.2 + 14.6 SD for vaginitis, with a mean overall
384 GEORGE R. PARKERSON JR et al.
suoguoo pue sas!nq
su!eqs pue su!eIds
atuo~puds pzsnedouab
awas!p Iu@oIoJnau Iaqlc
uoyaldaa
S!I!U!8t?~
u!ud qX?q MO-)
aseas!p lleaq yuays! 3!uo~q3
syqpeoapg
sasuas!p qwmo~s JaylO
sysnms
Lla!xuy
lapJos!p p!dg
asnqe 033uqoL
sawera
uO!Sua$Iad&
/
--000
000000
i mo-o-00
/ *0000000 /
moooooooo
1 -ooo-mo-00
~N000-0--00
-/o-00-0-0-000
/ w/o 0 0 0 0 0 -00-000
/
/
/
Duke Severity of Illness Checklist 385
Table 2. DUSOI diagnosis severity scores for the 21 most prevalent health problems (N = 414 patients with 803 problems)
*Duke Severity of Illness (DUSOI) scores for individual health problems, not including comorbid health problems. Scale: 0 = lowest severity, 108 = highest severity.
tSD = Standard deviation.
score for all health problems of 43.3 + 18.6 p = 0.003). They were not correlated signifi- SD. Overall severity scores showed weak cantly with race or marital status. positive correlations with older age (r = 0.26, Also shown in the table are the weighted p = 0.0001) and female gender (r = 0.12, DUSOI diagnosis scores used in calculating p = O.Ol), and weak negative correlations with the overall severity scores, and the percentage full-time employment (r = -0.20, p = 0.0001) contribution of each diagnosis score to overall and higher levels of education (r = -0.15, severity. For example, the sickest patients were
Table 3. Percentage contribution to DUSOI diagnosis severity score by the four severity parameter ratings for the 21 most prevalent health problems (N = 414 patients with 803 problems)
Severity parameters
Prognosis Symptom Complication without
level level treatment Treatability Health problems N W) (W W) W) Sprains and strains 8 43.5 1.9 20.0 Other neurological diseases 44.8 8.4 18.6 z Headache
*Duke Severity of Illness (DUSOI) overall severity scores, including all problems for each patient. Scale: 0 = lowest severity, 100 = highest severity.
tweighted severity score of the individual health problem used in calculating the overall severity score.
$Percentage contribution of the individual health problem weighted score to the overall severity score.
those whose diagnoses included depression. However, none of those patients had depression alone, and most of their overall severity was caused by their other health problems. The weighted diagnosis score for depression was only 10.9, and contributed only 18.0% of the overall severity score. This was true even though the mean unweighted DUSOI diagnosis score for depression alone was 32.6 + 10.7 SD, as shown in Table 2. That depression did not receive full weight in the calculation of overall severity indicates that the other individual comorbid diagnoses were rated as being more severe than depression in these 9 study patients. On the other hand, the 38 diabetic patients had a mean overall severity score of 56.9 + 16.8 SD, and 41.3% of the overall severity was due to diabetes.
DUSOI comorbidity scores were calculated, and for patients with the 21 most prevalent health problems, mean comorbidity scores ranged from a high of 54.8 + 12.7 SD for depression, to a low of 4.7 + 13.3 SD for sprains and strains. Patients with chronic health con- ditions generally had higher comorbidity scores than patients with acute illnesses. For example, mean comorbidity scores for chronic ischemic
heart disease (40.9 & 19.7 SD) and obesity (39.1 + 21.3 SD) were higher than mean comor- bidity scores for bruises and contusions (21.2 + 21.9 SD) and acute upper respiratory infection (16.4 f 18.0 SD). For all 803 health problems in the study population the mean comorbidity score was 32.0 + 24.3 SD.
The group of 54 patients who were studied longitudinally had a provider-generated mean DUSOI overall severity score of 47.9 & 15.8 SD at their initial visit and 47.1 & 16.0 SD at their return visit, with a test-retest correlation of 0.65 (p = 0.0001). The time interval was 75.5 f 52.4 SD days, with minimum of 7 days and a maxi- mum of 205 days. Overall, this test-retest group had a greater burden of illness than the other patients in the total study group, as indicated not only by their higher overall severity scores (47.9 compared with 42.7; t = 1.93, p = 0.0536), but also by their higher mean number of health problems per patient (2.43 at Time 1 and 2.46 at Time 2, compared with 1.87 for the other patients at Time 1; t = 2.74, p = 0.008). How- ever, both groups had similar diagnostic pro- files, i.e. they shared 8 of their 10 most prevalent health problems at the time of the initial office visit.
Duke Severity of Illness Checklist 387
When the health problems of the test-retest group at Time 1 were compared with those at Time 2, it was found that chronic problems, such as hypertension and diabetes, were preva- lent at both visits, and that acute problems such as headaches and sinusitis were less likely to be prevalent at the return visit. Review of the medical records of this group revealed that 69.8% were seen at the return visit for follow-up of chronic or recurrent conditions from the first visit, and that 30.2% were seen for acute prob- lems which appeared to be unrelated to the acute problems on the initial visit.
Comparison of provider and auditor scores Reliability analyses for the data which were
collected by retrospective chart audit using the DUSOI checklist showed that the intra-rater reliability coefficient for the principal auditor (No. 1) was 0.76 (F = 7.48) at a mean interval of 29.7 & 3.0 days for 67 charts with a mean DUSOI overall severity score of 39.1 &- 17.0 SD at Time 1 and 42.2 & 16.8 SD at Time 2. Intra- rater reliability for auditor No. 2 was 0.67 (I; = 5.05) at 31.7 f 4.2 days for 59 records. Inter-rater reliability between auditor No. 1 and auditor No. 2 was 0.47 (F = 2.74) for 59 records; between, No. 1 and No. 3 was 0.58 (F = 3.72) for 58 records; and between No. 2 and No. 3 was 0.63 (F = 4.44) for 58 records. The inter-rater reliability among the three auditors was 0.56 (F = 6.56) for 58 records. The p-value for all of these reliability coefficients was 0.000 1. The mean time required for auditing all 414 records by the principal auditor was 2.2 f 1.6 SD minutes per medical record, with a minimum of 1 minute or less and a maximum of 14 minutes.
Audit checklist data compared with provider- generated checklist data for the 414 study patients showed 859 health problems recorded by the auditor, compared with the 803 reported
by providers. There was no statistically signifl- cant difference (t = 1.61,~ = 0.11) between the mean number of diagnoses per patient by audit (2.07 f 1.26 SD) and the number by providers (1.94 + 1.15 SD). Likewise, the distribution of health problems per patient was not statistically different (&i-square = 6.58 with 6 df, p = 0.36). For example, the auditor reported 42.0% of patients having only one diagnosis compared with the provider report of 47.6%, and 30.2% of patients with two diagnoses compared with 26.8%.
Diagnostic lists generated by the auditor and the providers were similar. For example, 121 diagnoses of hypertension were reported by the auditor, compared with 116 by the providers. Of the patients with hypertension, 16 had hyper- tension alone by audit, compared with 17 by providers, and 26 had co-existing diabetes instead of the 24 reported by providers. For diabetes, 37 cases were reported by audit instead of 38, two of which had diabetes alone instead of the one reported by providers.
Assessment of the agreement between auditor and provider DUSOI scores showed a coefficient of agreement of 0.59 (p < 0.0001) between the audited overall severity scores (mean = 38.9 f 17.5 SD) and the provider over- all severity scores (mean = 43.3 f 18.6 SD). Coefficients of agreement between audited scores and individual provider scores for the three providers who evaluated 87.6% of the patients were 0.67, 0.55, and 0.53 (p <O.OOOl for all coefficients). Coefficients were calculated also by diagnosis for those five health problems listed by both the auditor and providers at a frequency of 20 or more. As shown in Table 5, coefficients of agreement for diagnosis severity scores were statistically significant except for lipid disorder, where the coefficient was only 0.17. Others ranged from the very weak coefficient of 0.19 for hypertension to the mod-
Table 5. Agreement of provider and auditor-generated DUSOI diagnosis and overall severity scores (N = 414 patients with 803 problems)
Health problems? N Provider Auditor ICC5 Provider Auditor ICC5
Hypertension 106 27.1 f 16.7 14.4 f 7.2 0.19**** 48.0 f 18.1 37.5 f 17.7 0.52**** Diabetes mellitus 35 35.0 f 21.9 25.2 f 18.3 0.70**** 57.7 f 17.1 46.5 f 16.7 0.61**** Tobacco abuse 25 31.3 + 14.0 20.3 f 8.3 0.47**** 45.9 z 18.0 40.0 ; 16.0 Lipid disorder 24 15.1 f 7.8 14.6 f 7.1 0.17**** 46.7 f 15.0 44.0 & 15.5 Bronchitis, acute 24 39.8 f 9.9 38.0 f 7.1 0.35. 47.8 f 13.8 46.5 f 10.7 0.37’
tHealth problems which were listed both by the providers and the auditor, and which had a frequency of 220. Patients with multiple diagnosis are included in the counts for each of their diagnoses.
$Mean f standard deviation. $ICC = Intraclass correlation coefficients between provider and auditor scores. ‘p < 0.05; **p < 0.01; ??**p < 0.001; ??***p < 0.0001.
388 GEORGE R. PARKERS~N JR et al.
erately strong coefficient of 0.70 for diabetes. Coefficients for overall severity scores varied from 0.37 for patients with bronchitis to 0.75 for tobacco abuse, and all were statistically significant.
Test-retest analyses for audited DUSOI overall severity scores showed a Spearman rank-order correlation coefficient of 0.59 (p = 0.0001). Comparisons between audited scores and providers’ scores were made to show the effect of having an auditor generate a score on one occasion and the provider at a different time, a strategy which might be useful in certain longitudinal clinical studies. For example, the provider might rate severity at the initial visit and an auditor, at the follow-up visit. The correlation was 0.44 (p = 0.001) between audited DUSOI at Time 1 and provider DUSOI at Time 2, and the correlation was 0.42 (p = 0.002) between provider DUSOI at Time 1 and audited DUSOI at Time 2.
Also, comparisons were made between the auditor and providers with regard to their use of the four severity parameters for generating severity scores. The distribution of the four severity parameter ratings was similar between auditor and providers, in that the symptom level ratings constituted 30.7% of the individual diagnosis DUSOI severity scores by audit, compared with 33.5% by providers. Compari- son of auditor with provider use of the other parameters was 6.4 vs 5.2% for complication level; 14.7 vs 23.6% for prognosis; and 48.2 vs 37.7% for treatability.
Patient morbidity also was assessed by the providers using an analog scale to indicate overall severity of illness. The analog overall severity scores for the 414 patients (mean: 37.0 + 24.2) showed a coefficient of agreement of 0.61 (p < 0.0001) with the DUSOI provider overall severity scores (mean: 43.3 f 18.6), and a coefficient of 0.42 (p < 0.0001) with the DUSOI audit checklist scores (mean: 38.9 + 17.5).
DISCUSSION
This study has attempted to quantify implicit judgments of medical providers in assessing the severity of illness of their patients. Although their judgments were focused on four explicit non-disease-specific parameters of severity, there were no explicit criteria for clinical ratings of severity within each parameter, and there were no explicit criteria for diagnosing the
patient health problems which are rated for severity. The result of this attempt to quantify qualitative data is a severity instrument which attains only a modest level of reliability by psychometric standards. However, while most of the coefficients of agreement for intra-rater, inter-rater, and provider-auditor severity scores do not fall into the “excellent” agreement category of greater than 0.75 suggested by Fleiss [22], they are within the “fair” agreement category of 0.40.75. Given the complexity of human health problems and their effect on people, the many variations in diagnostic labeling, and the unknowns of prognosis in terms of natural history of illness and expected response to therapy, the degree of reliability attained by the DUSOI in the present study is noteworthy.
There is no question that reliability can be improved by refining the DUSOI with specific diagnostic and severity criteria, and perhaps this should be done for studies of a limited number of health problems. In the meanwhile this simple instrument may be a reasonable tool for measuring severity from the perspective of the clinician who is active in the practice setting. The present study has demonstrated that the DUSOI Checklist can be used in the primary care office both by clinical providers at the time of the patient visit, and by auditors retro- spectively, to produce measures of severity for both individual and aggregated health prob- lems. All of a person’s health problems can be included in the process and contribute to the diagnosis, comorbidity, and overall severity scores, whether or not those problems have been defined as specific diseases.
Until medical knowledge advances to the point at which detailed algorithms based on sound scientific data have been developed for the course and management of most health problems, medicine will remain heavily depen- dent on implicit clinical judgment to determine severity of illness. This dependence upon clinical judgment is important, particularly when severity is being assessed for individual patients to measure severity as an outcome of their medical care. Perhaps clinical judgment is less essential when severity is assessed as an indicator of the intensity of medical care for determination of health policy or reimburse- ment for clinical services, as in the ambulatory case-mix measures.
Determination of “how sick the patient is” is one of the most important decisions for the
Duke Severity of Illness Checklist 389
clinician in providing health care for the patient. This is a judgment that often requires an enor- mous amount of data, clinical experience, basic knowledge of medicine, and information on how the particular patient behaves in response to his or her health problems. Is the patient symptomatic, and if so how severe are the symptoms? Have complications developed, and if so, of what magnitude? Is the condition life-threatening and/or disabling, and if so, to what extent? Is treatment indicated, and if so, what is the expected response to treatment? These basic questions, which must be answered in the clinical assessment of severity, provide a solid basis for conceptualization of the four severity parameters of the DUSOI.
heart disease (21.8%). Conversely, prognosis
The severity parameters for symptoms, prog- nosis, and treatability were used frequently by the providers in this study to determine severity. On the other hand, they used the complication parameter for determination of only 5.2% of the total severity of diagnosis severity scores. This may reflect the DUSOI characteristic of limiting evaluation to a l-week period of time, which might exclude complications which arose in the past few months. Also, it may result from this study population’s characteristic of gener- ally low prevalence of severe illnesses. Unless studying the DUSOI in patients with severe illness also shows rare use of the complications parameter, this low utilization of complications does not provide sufficient grounds for remov- ing this parameter from the DUSOI, primarily because of the known clinical importance of complications in very sick patients. In patient populations with a higher case mix of severe illness the complication parameter has been shown to be an important parameter for instru- ments which have been developed in those settings [ 1-1.
The data of this study indicate that acute illnesses which may respond well to therapy can generate diagnosis severity scores equal to or higher than some of the chronic illnesses which are usually considered more serious. This is explained by the higher contribution of symp- tom level parameter ratings than prognosis ratings for the acute illnesses at the time of the patient’s office visit. For example, in Table 2 the diagnosis severity score for sprains and strains (43.0) is actually higher than that for chronic ischemic heart disease (37.1). As seen in Table 3, symptom level ratings contributed more to the diagnosis severity score for sprains and strains (43.5%) than for chronic ischemic
ratings contributed more to the severity score for chronic ischemic heart disease (42.2%) than for sprains and strains (20.0%). The result for patients at the time of the office encounter was that these two very different health problems resulted in a high and almost equal burden of illness. The DUSOI is designed to quantify the total burden of illness for the patient at one point in time. These scores are consistent with that approach.
Also, this study demonstrates the clinical phenomenon of wide variation of severity within diagnostic categories. Diagnostic labels themselves give only partial information regard- ing severity. For example, DUSOI diagnosis severity scores ranged from a minimum of 6.3 to a maximum of 87.5 units for diabetic patients and from 0 to 75.0 units for hypertensives. This variability is a reality which clinicians must consider in patient management everyday, and which has been important in the development of disease-specific severity measures such as those using disease staging [24].
Wide variations in the contribution of a par- ticular health problem to overall severity also are demonstrated in this study. For example, diabetes contributed only 41.3% of overall severity for the 38 diabetics in this study group when measured by the DUSOI formula, which gives full weight to the diagnosis with the highest diagnosis severity score, and progress- ively diminishing weights to the less severe diagnoses. The DUSOI approach of quantitat- ive combination of diagnosis severity scores by weights makes clinical sense in that the patient’s overall burden of illness at a given point in time is usually predominated by the health problem of greatest severity at that time. For example, the severity of an acute injury, such as multiple trauma from an automobile accident, can over- shadow the severity of a well-controlled serious chronic health problem, such as diabetes, even when severity parameters such as prognosis are considered. As illustrated with data from the present study, the diagnosis severity scores for sprains and strains contributed 90.0% of the mean overall severity score of 46.0 for patients with sprains and strains, in contrast to a contribution of 26% by the diagnosis severity scores for chronic ischemic heart disease to its patients’ mean overall score of 52.4, as shown in Table 4.
Both individual diagnosis severity scores and comorbidity severity scores are important,
390 GEORGE R. PARKERKIN JR et al.
especially when a particular illness is being studied, and the comorbidity of other illnesses is being controlled for. For example, Coulehan et al. [23] and Broadhead et al. [24] used audited DUSOI scores in separate studies of depression to control for the severity of their patients’ illnesses other than depression. They calculated their comorbidity severity score by the same method described in this paper, i.e. by excluding the diagnosis severity score for depression from the scoring equation to obtain a depression-free comorbidity score.
The comparative data between severity scores generated by patient providers and the medical record auditor in the present study gave support for the validity of the DUSOI. Provider and auditor lists of health problems were very much alike, and the coefficients of agreement between provider and auditor overall severity scores were fairly high and statistically significant (ICC = 0.59, p < 0.0001). However, in some instances there were considerable differences between provider and auditor diag- nosis severity scores, such as that between the provider diagnosis severity scores for hypertension (mean = 27.1 +_ 16.7 SD) and the auditor scores (mean = 14.4 + 7.2 SD), shown in Table 5. Perhaps, part of this difference can be attributed to inadequacies in traditional medical record keeping, where symptomatology and complications are recorded more routinely than prognosis. While the provider and the patient were literally face-to-face at the time of severity assessment, the auditor had to glean information from the provider’s progress notes retrospectively. The most valid assessment in this situation may be that of the provider, rather than the auditor, because the provider may have much more information about the patient encounter than was recorded in the medical record. There are serious implications here for medical record keeping, particularly if records are to be used to measure severity of illness as an outcome of medical care. Records would be improved if providers included in their routine progress notes their patients’ prognosis and expected response to treatment, in addition to symptom and complication levels.
The test-retest analyses in this study are interesting because longitudinal data are very important in studying the sensitivity of a measure to changes in severity over time. In the group of 54 patients who consented to partici- pate in this study at the time of their initial and follow-up visits, the mean scores were
essentially the same at both visits, with a fairly high correlation coefficient of 0.65 (p c 0.0001). Although most of the return visits were for chronic health problems which may change little in severity short-term, one might expect that the severity would decrease significantly for many illnesses several weeks after an initial medical visit in response to treatment, natural history of illness, and/or the phenomenon of mathematical regression toward the mean. A much larger group of patients would be needed to study sensitivity of the DUSOI to change in severity over time, because of the wide variety of health problems in this type of population, the wide range of patients’ severity scores, the diverse reasons for their return visits, and the wide range of time between their initial and follow-up encounters. All of these factors decrease the precision of the present follow-up data, and increase the necessity for a larger sample size.
The present study is very limited in its ability to validate the DUSOI. There is no gold stan- dard severity instrument with which to compare the measure, and the collection of objective pathologic and physiologic data reflecting the severity of multiple illnesses was not practical as part of the study. Further research is needed with larger numbers of patients having each health problem, more objective severity data, use of same-day ratings by more than one provider, and long-term follow-up to determine validity of the DUSOI in terms of the poss- ible predictive effect of its scores upon future severity of illness.
Also, more research is needed on the validity of the DUSOI analog scale, the scores of which showed a fairly high and statistically significant agreement (ICC = 0.61,~ < 0.0001) with overall severity scores calculated from individual diagnosis severity scores. While the analog method provides a quick assessment of total burden of illness, it lacks the advantage of providing separate severity scores for each of the patient’s health problems. Even so, in the present study the DUSOI analog scores provided a form of internal validation for the DUSOI overall severity scores in that they required a somewhat different cognitive process by the clinician raters.
The DUSOI provides a methodology with fairly high reliability and with sufficient clinical face validity to justify further investigation of its use in medical outcomes assessment. Further research is needed to support its reliability and
Duke Severity of Illness Checklist 391
validity and to evaluate its ability to measure lo* changes in severity over time. The DUSOI 1 1. should prove useful as a method of controlling for severity in studies of functional health and quality of life, controlling for comorbidity in
12 *
studies of specific illnesses, and indicating the outcome of medical care in terms of whether or not severity of illness has been reduced by 13’ medical intervention.
Acknowledgements~ompletion of the DUSOI Check- 14* lists at the time of patient visits was performed by W. E. Broadhead, M.D., Ph.D., James W. R. Harding, III, M.D., M.P.H., Janet Jexsik, P.A.-C., Todd Shapley-Quinn, 15. M.D., and Bret C. Williams, M.D., M.P.H. Chart audits for reliability analyses were performed by Anthony Geraci, M.D. and Jonathan Sheline, M.D., M.S. Funding was provided by Glaxo, Inc., Research Triangle Park, NC, and 16. the Department of Community and Family Medicine. Duke University Medical Center.
1.
2.
3.
4.
5.
6.
I.
8.
9.
REFERENCES 17.
Kaplan MH, Feinstein AR. The importance of classi- fying initial co-morbidity in evaluating the outcome of diabetes mellitus. J Chron Dla 1974; 27: 387404. 18. Gonnella JS, Goran MJ. Quality of patient care-A measurement of change: the staging concept. Med care 1975; 13: 467473. Gonnella JS, Louis DZ, McCord JJ. The staging concept--an approach to the assessment of outcome of ambulatorv care. Med Care 1976: 14: 13-21. 19. Gonnella JS, Hornbrook MC, Louis DZ. Staging of disease, a case-mix measurement. JAMA 1984; 251: 637644. 20. Horn SD. Measuring severity of illness: Comparisons across institutions. Am J Public Health 1983; 73: 25-31. 21. Horn SD, Buckley G, Sharkey PD et al. Interhospital dilferences in severity of illness. N Engl J Med 1985; 313: 20-24. 22. Horn SD, Sharkey PD, Buckle JM ef al. The relation- ship between severity of illness and hospital length of 23. stay and mortality. Med Care 1991; 29: 305-317. Charlson ME, Sax FL, MacKenzie CR et al. Assessing illness severity: Does clinical judgement work? J Chron Dls 1986; 39: 439452. 24. Pompei P, Charlson ME, Douglas RG Jr. Clinical assessments as predictors of one year survival after hospitalization: Implications for prognostic stratifica- tion. J Clln Epldemlol 1988; 41: 275-284.
Greenfield S, Aronow HU, Elashoff RM ef ol. Flaws in mortality data. JAMA 1988; 260: 2253-2255. Barsky AJ, Wyshak G, Klerman GL. Medical and psychiatric determinants of outpatient medical utilii- ation. Med Care 1986: 24: 548-560. Parkerson GR Jr, Michener JL, Wu LR et al. Associ- ations among family support, family stress, and per- sonal functional health status. J Clln Eu&mlol 1989: 42: 217-229. Fetter RB, Averill R, Lichtenstein JL et al. Ambulat- ory Visit Groups: A framework for measuring pro- ductivity in ambulatory care. He&h Ben Res 1984; 19: 415-437. Horn SD, Buckle JM, Carver CM. Ambulatory Sever- ity Index: Development of an ambulatory case mix system. J Ambulatory Care Manage 1988; 11: 53-62. Tenan H, Fillmore H, Caress B et al. PACs: Classify- ing ambulatory care patients and services for clinical financial management. J Ambulatory Care Manage 1988; 11: 36-53. Averill R, Goldfield N, McGuire T et al. Design and evaluation of a prospective payment system for ambulatorv care. MCFA Contract 17-C-99 369/l-021. Wallingfo;d, CX‘ 3-M Health Information Systems; 1990. Startield B, Weiner J, Mumford L et al. Ambulatory Care Groups: A categorization of diagnoses for research and management. Health Serv Res 1991; 25: 990-1015. Weiner JP. Ambulatory case-mix methodologies: Application to Primary Care Research. Ceof Proc Prlmuy Care Reaearehr Theory and Methods. U.S. Department of Health and Human Services, Public Health Service, Agency for Health Care Policy and Research; September 1991. Parkerson GR Jr, Broadhead WE, Tse C-KJ. Quality of life and functional health of primary care patients. J cull Epldssnlol 1992; 45: 1303-1313. Intematlonal CIamlBeatlou of Health Problems ia Prl- mary Care. Chicago: American Hospital Association; 1975. Shrout PE, Fleiss JL. Intraclass correlation: uses in assessing rater reliability. Psycho1 Bull 1979; 86: 420428. Fleiss JL. Statisticd Methods for Rates and Pro- portiom, 2nd edn. New York: Wiley; 1981. Coulehan JL, Schulberg HC, Block MR et al. Medi- cal comorbidity of major depressive disorder in a nrimarv medical practice. Arch Intern Med 1990; 150: 2363-2367. - Broadhead WE, ClappChanning NE, Finch JN et al. Effects of medical illness and somatic svmntoms on treatment of depression in a family medicine residency practice. Gen Heap Psycblatry 1989; 11: 194-200.
(Appendices overleaf)
392 GEORGE It. PARKERSON JR et al.
APPENDIX A
Duke Severity of Illness (DUSOI) Checklist’ ID Number:
Provider:
If Audit, Date: Auditor:
Date of Encounter:
Patient’s Name:
Diagnosis or Health Problem:
None Questionable Mild Mederate Major
I. Symptoms (past week): Cl6 01 02 03 04
2. Complications (past week): Cl6 01 02 cl3 04
Disability
3. Prognosis (next 6 months, None Mild Moderate Major Threat to Life
without treatment): 06 Cl1 cl2 cl3 04
Need for Treatment Expected Response to Treatment
No Questionable I
If Yes - Good Questionable Poor
4. Treatability: 00 01 02 03 04
Diagnosis or Health Problem:
None Questionable Mild Moderate Major
I. Symptoms (past week): Cl6 01 02 03 04
2. Complications (past week): Cl6 01 02 ??3 ??4
Disability
3. Prognosis (next 6 months, None Mild Moderate Major Threat to Life
without treatment): Cl6 01 02 cl3 04
Need for Treatment Expected Response to Treatment
No Questionable I
If Yes - Good Questionable Poor
4. Treatability: cl0 01 Cl2 03 04
Diagnosis or Health Problem: _
None Questionable Mild Moderate Major
1. Symptoms (past week): 06 01 02 03 04
2. Complications (past week): 06 01 cl2 03 cl4
Disability
3. Prognosis (next 6 months, None Mild Moderate Major Threat to Life
without treatment): 176 01 cl2 03 cl4
Need for Treatment Expected Response to Treatment -
No Questionable III
If Yes - Good Questionable Poor
4. Treatability: cl0 01 02 03 04
USE ADDITIONAL SHEETS FOR OTHER DIAGNOSES
*Copyright 1990 Department of Community and Family Medicine, Duke University Medical Center, Durham, NC, U.S.A.
Duke Severity of Illness Checklist 393
APPENDIX B
Patient’s Name:
Provider:
ID Number:
Date of Encounter:
Duke Severity of Illness Analog Scale (DUSOI-A)”
(Patient’s overall severity of illness as assessed by the provider)
Instructions:
Please mark with an X the appropriate place along the line below to indicate how you would rate this patient’s overall
severity of illness during the pest week.
Lowest severity applies to someone whose total set of diagnoses results in the fewest symptoms and complications,
the least disability and threat to life, the least need for treatment, and the best expected response to treatment if needed.
Highest severity applies to someone whose total set of diagnoses results in the most symptoms and complications,
the most disability and greatest threat to life, the most need for treatment, and the worst expected response to
treatment.
LOWEST SEVERITY
(Please mark one X along the line)
HIGHEST SEVERITY
How confident are you that your rating of this patient’s severity of illness is accurate? Please circle the appropriate category.
Not at all Confident
0
Not Very Confident
1
Very Confident
2
Absolutely Confident
3
*Copyright 1990 Department of Community and Family Medicine, Duke University Medical Center, Durham, NC, U.S.A.
