Derivation and Validation of AutomatedElectronic Search Strategies to Extract CharlsonComorbidities From Electronic Medical RecordsBalwinder Singh, MBBS; Amandeep Singh, MBBS; Adil Ahmed, MBBS;Gregory A. Wilson, BS; Brian W. Pickering, MB, BCh, FFARCSI;Vitaly Herasevich, MD, PhD; Ognjen Gajic, MD, MSc; and Guangxi Li, MD
Abstract
Objective: To develop and validate automated electronic note search strategies (automated digital algorithm) to iden-tify Charlson comorbidities.Patients and Methods: The automated digital algorithm was built by a series of programmatic queries applied to aninstitutional electronic medical record database. The automated digital algorithm was derived from secondary analysisof an observational cohort study of 1447 patients admitted to the intensive care unit from January 1 through December31, 2006, and validated in an independent cohort of 240 patients. The sensitivity, specificity, and positive and negativepredictive values of the automated digital algorithm and International Classification of Diseases, Ninth Revision (ICD-9)codes were compared with comprehensive medical record review (reference standard) for the Charlson comorbidities.Results: In the derivation cohort, the automated digital algorithm achieved a median sensitivity of 100% (range,99%-100%) and a median specificity of 99.7% (range, 99%-100%). In the validation cohort, the sensitivity of theautomated digital algorithm ranged from 91% to 100%, and the specificity ranged from 98% to 100%. The sensitivityof the ICD-9 codes ranged from 8% for dementia to 100% for leukemia, whereas specificity ranged from 86% forcongestive heart failure to 100% for leukemia, dementia, and AIDS.Conclusion: Our results suggest that search strategies that use automated electronic search strategies to extract Charl-son comorbidities from the clinical notes contained within the electronic medical record are feasible and reliable.Automated digital algorithm outperformed ICD-9 codes in all the Charlson variables except leukemia, with greatersensitivity, specificity, and positive and negative predictive values.
FmR(BDCAscVtHChinese Medical Science,Beijing (G.L.).
C omorbidity is defined as any distinct clini-cal entity that preexists or occurs during apatient’s primary disease.1 Various studies
have documented the role of comorbidities in pre-dicting a patient’s outcome.2-6 The Charlson Co-morbidity Index (CCI) was developed to estimatethe long-term (1-year) mortality of patients admit-ted to the hospital or enrolled in research studies onthe basis of the comorbid conditions.7 The CCI con-sists of 19 comorbid conditions, and each comor-bidity is assigned a score of 1, 2, 3, or 6 based on therelative risk of 1-year mortality.7 The CCI has beenvalidated in several different populations and iswidely used in various health services research andcritical care studies.3,8-10
The use of electronic medical records (EMRs) isincreasing, and these records are used not only inclinical practice but also in most epidemiologic andhealth care research. Recent mail survey findingsfrom the National Ambulatory Medical Care Surveyconducted by the Centers for Disease Control andPrevention reported an increase in the adoption of
As a part of the current health system reform in theUnited States, the government has invested largesums of money to support and promote the adop-tion of the EMR system in the country. The 2009Health Information Technology for Economic andClinical Health Act provides possible incentives forhospitals to implement EMR systems; hence, mostphysicians and hospitals intend to implement EMRsystems within the next few years.12,13 Tradition-lly, the CCI was identified and calculated solelysing the manual medical record review. In 1992,eyo et al14 developed an electronic application toolased on International Classification of Diseases, Ninthevision (ICD-9) codes to automatically calculate theCI.14 Currently, this method is applied in most
research projects for baseline comorbidities adjust-ments,5,10 although the literature has reported var-ious concerns about the accuracy and underreport-ing of comorbidities using ICD-9 codes.15-21
With the growing notion of EMRs as a tool toeduce cost and improve safety,22 adoption of EMR
ystems in US hospitals is steadily increasing. The
g/10.1016/j.mayocp.2012.04.015dical Education and Research
or editorialomment, seeage 811
rom Multidisciplinary Epide-iology and Translationalesearch in Intensive CareB.S., A.S., A.A., G.A.W.,.W.P., V.H., O.G., G.L.),ivision of Pulmonary andritical Care Medicine (B.S.,.A., O.G., G.L.), and Divi-
ion of Critical Care Medi-ine (A.S., G.A.W., B.W.P.,.H.), Mayo Clinic, Roches-
information overload becomes a hindrance to theeffective use of EMRs, which may lead to reductionin performance and alter patient safety.23 Thus, notonly is implementation of an automated electronicnote search strategy to identify a patient’s baselinecomorbidities useful in medical research, but earlyidentification of the comorbidities from the EMRmight also be helpful for the efficient treatment ofpatients.
We therefore aimed to develop and validate anautomatic note search strategy (automated digitalalgorithm) based on EMR notes to identify CCI co-morbidities. Our secondary aim was to compare thesensitivity, specificity, positive predictive value(PPV), and negative predictive value (NPV) of auto-matic note search strategy and ICD-9 code searchwith comprehensive medical record review (refer-ence standard) in detecting CCI comorbidities fromthe EMR.
PATIENTS AND METHODSThe study was approved by the Mayo Clinic Institu-tional Review Board, for the use of medical recordsfor research.
Study PopulationThe derivation cohort was from a community-based, retrospective study of 1447 eligible patients(patients �18 years of age who gave research autho-rization) admitted to the intensive care unit (ICU)during 2006 from Olmsted County, Minnesota.24
The automated digital algorithm was further vali-dated against a randomly selected cohort of 240 pa-tients who gave research authorization from a retro-spective cohort of 651 patients with severe sepsisadmitted to the ICU from January 2007 to Decem-ber 2009,25 using JMP statistical software (version9.0, SAS Institute Inc, Cary, NC). All those whodenied research authorization were excluded.
Data Extraction StrategiesAll the CCI comorbidities in the 5-year interval be-fore the time of admission were manually collectedby trained research fellows according to the pub-lished definitions by Charlson et al.7 Each recordwas reviewed by multiple reviewers. We comparedthe 2 data extraction strategies: (1) CCI scores ex-tracted by the automated digital algorithm and (2)CCI scores extracted by using ICD-9 codes.
Automated Electronic Note Search Strategies(Automated Digital Algorithm)The development of an automated electronic notesearch strategy (automated digital algorithm) re-
quires considerable investment in both time and ex- t
Mayo Clin Proc. � September 2012;8
pertise. This retrospective study used data from theMayo Clinic Life Sciences System (MCLSS). TheMCLSS database is the single centralized databasefor all the Mayo clinic hospital data. The MCLSS isan exhaustive clinical data warehouse that stores pa-tient demographic characteristics, diagnoses, andhospital, laboratory, flow sheet, clinical, and patho-logic data gathered from various clinical and hospi-tal source systems within the institution. TheMCLSS encompasses a near real-time model ofMayo’s EMR system.26 We used a Data Discoveryand Query Builder (DDQB) tool set to access thedata contained within the MCLSS database. TheDDQB can search for demographic characteristics,clinical data, hospital admissions information, diag-nosis codes, procedure codes, laboratory test re-sults, flow sheet data, pathology reports, and geneticdata. A valid institutional review board number isneeded to retrieve patient data, which can be usedfor research using the MCLSS. The DDQB providesa unique text search strategy by which researcherscan rapidly search for distinct words or entities inthe EMR system.
The DDQB is based on Boolean logic to createfree text searches.26 All the free text searches wererun independently by 2 physician investigators (A.S.and B.S.). To initiate a query, we entered all thesynonyms, abbreviations, and the most commonsymptoms associated with the comorbidity. In addi-tion, we excluded the negative terms mentioned inthe clinical notes to make the automated digital al-gorithms more specific (see Supplemental Appendix1 for the list of excluded terms; available online athttp://www.mayoclinicproceedings.org). For theextraction of all the CCI comorbidities, the auto-mated digital algorithm explored the EMR of eacheligible patient during the 5 years before the date ofadmission in the medical and surgical history sec-tion of the EMR. For better understanding, an auto-mated digital algorithm for peptic ulcer disease isshown in the Figure. The automated digital algo-rithm for CCI comorbidities were continuouslyrefined by adding or excluding terms to improvethe sensitivity and specificity to 95% or more. Thefinal search terms used for building the auto-mated digital algorithm are shown in Supplemen-tal Appendix 2 (available online at http://www.
ayoclinicproceedings.org). To validate the auto-ated digital algorithm, the sensitivity and specific-
ty were calculated against the reference standard ofomprehensive medical record review.
ICD-9–Based Data Extractionhe MCLSS administrative database was used to cal-ulate the ICD-9 coded CCI comorbidities according
Manual Data Extraction (Reference Standard)Manual data extraction is the traditional method ofascertaining comorbidity data from clinical notes.Trained research fellows manually collected comor-bidity data according to the definitions published byCharlson et al.7 Comorbid conditions are mostly re-corded in the medical and surgical history section ofthe clinical notes. So to maintain uniformity andefficiently identify specific comorbid conditions,only the medical and surgical history sections of theclinical notes were ascertained. If the comorbid con-
Refine the search
No
Final q
Peptic ulcer discompare it with re
Sensitivity, s≥95
Clinical
Note date <5 y fromthe date of admission
Keyw
peptic ulcer disease
duodenalulcer%
gastrophagulce
PUOR
OR
FIGURE. The automated digital algorithm for sea
dition was not identified in this particular section of
Mayo Clin Proc. � September 2012;87(9):817-824 � http://dx.doi.orwww.mayoclinicproceedings.org
the EMR, it was assumed to be negative. Researchfellows involved in manual data extraction weremasked to the automated electronic note searchstrategy results.
Statistical AnalysesSensitivity and specificity of both the automateddigital algorithm and ICD-9 codes search were cal-culated based on comparisons of the test results andthe reference standard in the 2 cohorts. The PPV and
The 95% confidence intervals were calculated usingan exact test for proportions. JMP statistical software(version 9.0, SAS Institute Inc) was used for all dataanalysis.
RESULTSDuring the study, 1447 consecutive eligible patientsadmitted to the ICU from January 1 through Decem-ber 31, 2006, at Mayo Clinic, Rochester, MN, wereincluded in the derivation cohort. The demographiccharacteristics and baseline comorbidity status ofthe derivation and validation cohort are summa-rized in Table 1. The most prevalent comorbiditieswere chronic lung diseases (24.3%) and diabetesmellitus (24.3%) in the derivation cohort and ma-lignant tumor (32.1%) in the validation cohort. Inthe derivation cohort, the automated digital algo-rithm achieved a median sensitivity of 100% (range,99%-100%) and a median specificity of 99.7%(range, 99%-100%). A summary of the prevalenceof CCI comorbidities, sensitivity, and specificity ofautomated digital algorithm in the derivation andvalidation cohorts compared with the referencestandard is given in Table 2. In addition, the con-cordance and discordance (true-positive, true-neg-ative, false-positive, and false-negative results) be-tween the automated digital algorithm and referencestandard in the derivation and validation cohorts aresummarized in Table 2.
Table 3 summarizes the sensitivity, specificity,PPV, and NPV of the 2 data extraction strategies (au-tomated digital algorithm and ICD-9 codes search) forall CCI comorbidities in the validation cohort. The sen-sitivity for identifying CCI comorbidities using the au-tomated digital algorithm ranged from a minimum of91% for lymphoma to a maximum of 100% for con-gestive heart failure (median, 98.5%; interquartilerange [IQR], 94%-100%). Sensitivities for extractingcomorbidities using ICD-9 codes ranged from a mini-mum of 8% for dementia to 100% for leukemia (me-dian, 66%; IQR, 51%-76%). The automated digital al-gorithm achieved a median specificity of 99.6% (IQR,99%-100%) compared with 97% (IQR, 91%-98%) for
aracteristics and Baseline Comorbidity Status of theohorts
Derivation cohort(n�1447)
Validation cohort(n�240)
66 (49-80) 69 (58-80)
756 (52.2) 129 (53.8)
1312 (90.7) 217 (90.4)
982 (67.9) 204 (85.0)
ICD-9 codes.
Mayo Clin Proc. � September 2012;8
In different CCI comorbidity domains, the PPVanged from 80% to 100% for the automated digitallgorithm, with the lowest being hemiplegia and theighest connective tissue disease, leukemia, lym-homa, metastasis, mild liver disease, peptic ulcerisease, and peripheral vascular disease (100%).he PPV for ICD-9 codes ranged from 0% to 100%,ith the highest being leukemia and lowest beingIDS (the ICD-9 codes from Deyo et al for AIDS, ie,42.x-044.x take into account patients with human
mmunodeficiency virus and patients with AIDS).he CCI index weight of 6 is given only to AIDSatients and the only patient detected as havingIDS using the ICD-9 code in the validation cohortas a false-positive result, hence the PPV of zero.he NPV ranged from 97% to 100% for the auto-ated digital algorithm compared with 79% to
00% for ICD-9-Clinical Modification codes. The me-ian PPV and NPV were 97.2% (IQR, 94-100%) and9.6% (IQR, 99-100%) for the automated digitallgorithm and 62.5% (IQR, 42%-78%) and 97.2%IQR, 91%-99%) for the ICD-9 codes, respectively.
ISCUSSIONesults of this study suggested the feasibility andalidity of the automatic note search strategy indentifying CCI comorbidities in the EMR. Our re-ults indicate that the sensitivities of the automaticote search strategy were considerably better than
CD-9 codes to search for all the CCI variables ex-ept one, leukemia, for which the sensitivity of bothearch strategies was 100%. The specificity and thePV of the automatic note search strategy were also
qual or superior to the ICD-9 codes search in all theCI comorbidities. In addition, our results con-rmed the findings of previous studies on the reli-bility and accuracy of electronic search; for exam-le, Alsara et al26 also reported that electronic query
resulted in accurate and highly efficient dataextraction.
The CCI is being widely used by health careresearchers to predict short-term (30 days) andlong-term (1 year) mortalitiy in ICU patients.27-30
To compare the meaningful differences in patients’outcomes, it is essential to balance the baseline co-morbidity conditions. The CCI is one of the mostcommonly used tools to measure the baseline co-morbidities before ICU admission. A recent studyperformed by Christensen et al31 discussed the im-
ortant role of the CCI combined with administra-ive data in predicting short- and long-term mortal-ty for ICU patients. Although D’Hoore et al32
described the CCI index as a resourceful way to per-form risk adjustment from administrative databases,Poses et al4 reported enhanced discrimination of in-patient mortality using the CCI index. Currently, an
TABLE 1. Demographic ChDerivation and Validation C
TABLE 2. Prevalence of Charlson Comorbidities, Sensitivity, Specificity, and the Concordance and Discordance Between the AutomatedDigital Algorithm and Reference Standard in the Derivation Cohort and Validation Cohort
Charlson variable CohortNo. ofpatients
Prevalence(%)
Sensitivity (%)(95% CI)
Specificity (%)(95% CI)
True-positiveresult (No.)
True-negativeresult (No.)
False-positiveresult (No.)
False-negativeresult (No.)
AIDS DC 2 0.1 100 (20-100) 100 (99-100) 2 1445 0 0
cally extract CCI comorbidities.10,33,34 However,the ICD-9–coded administrative databases lack aclinical definition for diagnoses, causing variabilityin coding practices.35 Our results revealed thatICD-9 codes underreport the comorbidities thatsubstantiate the finding of the previous stud-ies.19,20,36,37 The underreporting could be attribut-able to extra emphasis on the procedures and compli-cations on admission, compared with the comorbidities,for monetary reasons.20 Romano et al38 also foundthat the CCI comorbidities were not accurately de-fined in ICD-9 codes, which produced interobservervariations in ICD-9 codes assigned to the comorbidi-ties. Although automatic searches using ICD-9 codesto identify comorbidities has been used in many re-search projects, the lack of accuracy in criteria usedby the staff who code medical records may differfrom physicians’ criteria in diagnosing a medicalcondition, which significantly limits the broad useof this method. The automatic note search strategieswere derived from the algorithm-incorporated key-word and program for a query within the specificnote section. This approach enhanced the use of thepatient database query and tremendously reduced
he Automated Digital Algorithm and ICD-9 Code Search
Automated digital algorithm
Sensitivity (%)(95% CI)
Specificity (%)(95% CI) PPV (%) NPV
100 10
95 (82-100) 99 (96-100) 95 9
98 (90-100) 99 (96-100) 97 9
100 (90-100) 99 (96-100) 96 10
93 (67-100) 100 (98-100) 100 10
92 (62-100) 100 (97-100) 92 10
92 (81-97) 99 (96-100) 96 9
mage 100 (82-100) 99 (96-100) 89 10
100 (40-100) 100 (97-100) 80 10
100 (63-100) 100 (98-100) 100 10
91 (57-100) 100 (98-100) 100 10
96 (89-99) 99 (95-100) 97 9
100 (68-100) 100 (98-100) 100 10
100 (63-100) 100 (98-100) 100 10
se 100 (67-100) 99 (97-100) 83 10
se 99 (91-100) 98 (94-100) 96 9
97 (83-100) 100 (97-100) 97 10
95 (72-100) 100 (98-100) 100 10
100 (82-100) 100 (98-100) 100 10
� International Classification of Diseases, Ninth Revision; NPV � ne
the time when compared with the manual medical M
Mayo Clin Proc. � September 2012;8
record review (mean time taken to manually review1 patient note for the CCI comorbidities rangedfrom 5 to 10 minutes). The implementation of anelectronic strategy to extract information is not onlyuseful for research purposes but also may be helpfulfor the treatment of patients.39 Because an automatedigital algorithm provides accurate information aboutpatient’s comorbidities, it will help physicians to
ecognize comorbidity information early and mightelp in better treatment. Comorbidities act as a prog-osticating factor for patient survival and treatment-elated outcomes. Patients with higher CCI scoresre at increased risk for readmissions and hospital-zations; thus, using automated digital algorithms todentify comorbidities early will certainly be an im-ortant factor and might well be used for early pal-
iative consultations if needed in the future. Theigh sensitivity and specificity of the automated dig-
tal algorithm make it an important tool for physi-ians and investigators in accurately estimating co-orbidities and might help in making early
ecisions and avoiding medical errors.Another alternative search strategy to identify
though this method produced better performancethan the ICD-9 code search strategy, there were alsosignificant limitations for broad use in clinical re-search.40 Chiang et al41 suggested that SNOMED-CTcoding is imperfect and unreliable and requires physi-cian training and repeated testing. Furthermore,SNOMED-CT does not satisfactorily distinguish theexact terms at the clinical interface level for the studytemplate at the current stage.42
Our search strategies also had certain limita-tions. First, performance of the automated digitalalgorithm and coding of the CCI is dependent on thequality of the database and consistency of the textentries, which limits the applicability of this ap-proach to units with this database or one similar.However, the logic and the free text search conceptcould be generalized to other institutions; it pro-vides potential for diffusion of the method at siteswilling to replicate the programming effort becausethe medical documentation training is similar acrossthe country. Because electronic clinical notes are be-coming a standard feature of the modern era, ourapproach will become more generalizable. Second,we only focused on a pertinent section (medical andsurgical history) of clinical notes to search for co-morbidities, which might have caused us to misssome information provided in other note sections,although the same validation process can be ex-tended to other sections of clinical notes because theconcept remains the same. Third, the data can bemissed because of errors or corruption in the datawarehouse.43 However, this will only account for asmall proportion of the database. Fourth, some ofthe CCI comorbidities definitions are outdated.Since the original CCI was developed in 1987,medicine has undergone a vast amount of change.Certain diseases, such as AIDS, no longer have thesame relative risk of mortality as when the CCIwas developed. Similarly, criterion for untreatedthoracic and abdominal aneurysm 6 cm or largerfor diagnosis of peripheral vascular disease needsreassessment. The latest guidelines advocate sur-gery when the aneurysm is 5.5 cm or larger.44
However, we could refine our search strategy toidentify variables according to any new defini-tions by modifying the algorithm for new defini-tions. Finally, because of the retrospective natureof the study, we only included documented co-morbidity in the definite diagnostic criteria.
In conclusion, CCI comorbidities can be cor-rectly identified using the automated digital algo-rithm. The combination of good sensitivity, spec-ificity, and easy calculation should encouragephysicians to implement the automated digital al-gorithm in their clinical practice and medical
research.
Mayo Clin Proc. � September 2012;87(9):817-824 � http://dx.doi.orwww.mayoclinicproceedings.org
ACKNOWLEDGMENTSWe thank all members of the Multidisciplinary Ep-idemiology and Translational Research in IntensiveCare group for constant and constructive feedback.
SUPPLEMENTAL ONLINE MATERIALSupplemental material can be found online athttp://www.mayoclinicproceedings.org.
Abbreviations and Acronyms: CCI � Charlson Comor-idity Index; CI � confidence interval; DDQB � Data Dis-overy and Query Builder; EMR � electronic medical re-ord; ICD-9 � International Classification of Disease, Ninth
Grant Support: This work was supported by the NationalInstitutes of Health grant RC1 LM10468Z-01.
Correspondence: Address to Guangxi Li, MD, Division ofPulmonary and Critical Care Medicine, Department ofMedicine, Mayo Clinic, 200 First St SW, Rochester, MN55905 ([email protected]).
REFERENCES1. Feinstein AR. The pre-therapeutic classification of co-morbid-
ity in chronic disease. J Chronic Dis. 1970;23(7):455-468.
2. Cleves MA, Sanchez N, Draheim M. Evaluation of two com-peting methods for calculating Charlson’s comorbidity indexwhen analyzing short-term mortality using administrative data.J Clin Epidemiol. 1997;50(8):903-908.
3. de Groot V, Beckerman H, Lankhorst GJ, Bouter LM. How tomeasure comorbidity. a critical review of available methods.J Clin Epidemiol. 2003;56(3):221-229.
4. Poses RM, McClish DK, Smith WR, Bekes C, Scott WE. Pre-diction of survival of critically ill patients by admission comor-bidity. J Clin Epidemiol. 1996;49(7):743-747.
5. Durairaj L, Will JG, Torner JC, Doebbeling BN. Prognosticfactors for mortality following interhospital transfers to themedical intensive care unit of a tertiary referral center. CritCare Med. 2003;31(7):1981-1986.
6. Keenan SP, Dodek P, Chan K, et al. Intensive care unit admis-sion has minimal impact on long-term mortality. Crit Care Med.2002;30(3):501-507.
7. Charlson ME, Pompei P, Ales KA, MacKenzie CR. A newmethod of classifying prognostic comorbidity in longitudinalstudies: development and validation. J Chronic Dis. 1987;40(5):373-383.
8. Quan H, Li B, Couris CM, et al. Updating and validating theCharlson comorbidity index and score for risk adjustment inhospital discharge abstracts using data from 6 countries. Am JEpidemiol. 2011;173(6):676-682.
9. Brooks SE, Ahn J, Mullins CD, Baquet CR. Resources and useof the intensive care unit in patients who undergo surgery forovarian carcinoma. Cancer. 2002;95(7):1457-1462.
0. Needham DM, Scales DC, Laupacis A, Pronovost PJ. A sys-tematic review of the Charlson comorbidity index using Ca-nadian administrative databases: a perspective on risk adjust-
ment in critical care research. J Crit Care. 2005;20(1):12-19.
11. Hsiao C, Hing E, Socey C, Cai B. Electronic Health RecordSystems and Intent to Apply for Meaningful Use IncentivesAmong Office-based Physician Practices: United States, 2001-2011. National Center for Health Statistics (NCHS) DataBrief; No. 79: November 2011. http://www.cdc.gov/nchs/data/databriefs/DB79.pdf. Accessed March 2, 2012.
12. Bays RA, Kaelin LD. Electronic medical records for the office.J Vasc Surg. 2010;51(5):1302-1308.
13. Blumenthal D, Tavenner M. The “meaningful use” regulationfor electronic health records. N Eng J Med. 2010;363(6):501-504.
14. Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comor-bidity index for use with ICD-9-CM administrative databases.J Clin Epidemiol. 1992;45(6):613-619.
15. McCarthy EP, Iezzoni LI, Davis RB, et al. Does clinical evidencesupport ICD-9-CM diagnosis coding of complications? MedCare. 2000;38(8):868-876.
16. Kern EF, Maney M, Miller DR, et al. Failure of ICD-9-CM codesto identify patients with comorbid chronic kidney disease indiabetes. Health Serv Res. 2006;41(2):564-580.
17. Iezzoni LI, Foley SM, Daley J, Hughes J, Fisher ES, Heeren T.Comorbidities, complications, and coding bias: does the num-ber of diagnosis codes matter in predicting in-hospital mortal-ity? JAMA. 1992;267(16):2197-2203.
18. Romano PS, Roos LL, Luft HS, Jollis JG, Doliszny K; IschemicHeart Disease Patient Outcomes Research Team. A compar-ison of administrative versus clinical data: coronary arterybypass surgery as an example. J Clin Epidemiol. 1994;47(3):249-260.
19. Hawker GA, Coyte PC, Wright JG, Paul JE, Bombardier C.Accuracy of administrative data for assessing outcomes afterknee replacement surgery. J Clin Epidemiol. 1997;50(3):265-273.
20. Powell H, Lim LL, Heller RF. Accuracy of administrative data toassess comorbidity in patients with heart disease: an Australianperspective. J Clin Epidemiol. 2001;54(7):687-693.
21. Quan H, Parsons GA, Ghali WA. Validity of information oncomorbidity derived from ICD-9-CCM administrative data.Med Care. 2002;40(8):675-685.
22. Zlabek JA, Wickus JW, Mathiason MA. Early cost and safetybenefits of an inpatient electronic health record. JAMA. 2011;18(2):169-172.
23. Ahmed A, Chandra S, Herasevich V, Gajic O, Pickering BW.The effect of two different electronic health record userinterfaces on intensive care provider task load, errors of cog-nition, and performance. Crit Care Med. 2011;39(7):1626-1634.
24. Cartin-Ceba R, Kojicic M, Li G, et al. Epidemiology of criticalcare syndromes, organ failures and life support interventions ina suburban U.S. community. Chest. 2011;140(6):1447-1455.
25. Paturi A, Valerio-Rojas JC, Kashyap R, Sathick IJ, SubramanianA, Cartin-Ceba R. Statin therapy is associated with decreasedhospital mortality in patients with severe sepsis and septic shock[abstract]. Chest. 2011;140(4_MeetingAbstracts):1063A.
26. Alsara A, Warner DO, Li G, Herasevich V, Gajic O, Kor DJ.Derivation and validation of automated electronic searchstrategies to identify pertinent risk factors for postoperativeacute lung injury. Mayo Clin Proc. 2011;86(5):382-388.
27. Ho KM, Knuiman M, Finn J, Webb SA. Estimating long-termsurvival of critically ill patients: the PREDICT model. PLoS One.2008;3(9):e3226.
