From the SelectedWorks of Shinyi Wu

January 2006

Systematic Review: Impact of Health InformationTechnology on Quality, Efficiency, and Costs ofMedical Care

ContactAuthor

Start Your OwnSelectedWorks

Notify Meof New Work

Available at: http://works.bepress.com/shinyi_wu/6

Systematic Review: Impact of Health Information Technology onQuality, Efficiency, and Costs of Medical CareBasit Chaudhry, MD; Jerome Wang, MD; Shinyi Wu, PhD; Margaret Maglione, MPP; Walter Mojica, MD; Elizabeth Roth, MA;Sally C. Morton, PhD; and Paul G. Shekelle, MD, PhD

Background: Experts consider health information technology key toimproving efficiency and quality of health care.

Purpose: To systematically review evidence on the effect of healthinformation technology on quality, efficiency, and costs of healthcare.

Data Sources: The authors systematically searched the English-language literature indexed in MEDLINE (1995 to January 2004),the Cochrane Central Register of Controlled Trials, the CochraneDatabase of Abstracts of Reviews of Effects, and the PeriodicalAbstracts Database. We also added studies identified by experts upto April 2005.

Study Selection: Descriptive and comparative studies and system-atic reviews of health information technology.

Data Extraction: Two reviewers independently extracted informa-tion on system capabilities, design, effects on quality, system ac-quisition, implementation context, and costs.

Data Synthesis: 257 studies met the inclusion criteria. Most studiesaddressed decision support systems or electronic health records.

Approximately 25% of the studies were from 4 academic institu-tions that implemented internally developed systems; only 9 studiesevaluated multifunctional, commercially developed systems. Threemajor benefits on quality were demonstrated: increased adherenceto guideline-based care, enhanced surveillance and monitoring, anddecreased medication errors. The primary domain of improvementwas preventive health. The major efficiency benefit shown wasdecreased utilization of care. Data on another efficiency measure,time utilization, were mixed. Empirical cost data were limited.

Limitations: Available quantitative research was limited and wasdone by a small number of institutions. Systems were heteroge-neous and sometimes incompletely described. Available financialand contextual data were limited.

Conclusions: Four benchmark institutions have demonstrated theefficacy of health information technologies in improving quality andefficiency. Whether and how other institutions can achieve similarbenefits, and at what costs, are unclear.

Ann Intern Med. 2006;144:E-12-E-22. www.annals.orgFor author affiliations, see end of text.

Health care experts, policymakers, payers, and consum-ers consider health information technologies, such as

electronic health records and computerized provider orderentry, to be critical to transforming the health care industry(1–7). Information management is fundamental to healthcare delivery (8). Given the fragmented nature of healthcare, the large volume of transactions in the system, theneed to integrate new scientific evidence into practice, andother complex information management activities, the lim-itations of paper-based information management are intu-itively apparent. While the benefits of health informationtechnology are clear in theory, adapting new informationsystems to health care has proven difficult and rates of usehave been limited (9–11). Most information technologyapplications have centered on administrative and financialtransactions rather than on delivering clinical care (12).

The Agency for Healthcare Research and Qualityasked us to systematically review evidence on the costs andbenefits associated with use of health information technol-ogy and to identify gaps in the literature in order to pro-vide organizations, policymakers, clinicians, and consumersan understanding of the effect of health information tech-nology on clinical care (see evidence report at www.ahrq.gov). From among the many possible benefits and costs ofimplementing health information technology, we focus

here on 3 important domains: the effects of health infor-mation technology on quality, efficiency, and costs.

METHODS

Analytic FrameworksWe used expert opinion and literature review to de-

velop analytic frameworks (Table) that describe the com-ponents involved with implementing health informationtechnology, types of health information technology sys-tems, and the functional capabilities of a comprehensivehealth information technology system (13). We modified aframework for clinical benefits from the Institute of Med-icine’s 6 aims for care (2) and developed a framework forcosts using expert consensus that included measures such asinitial costs, ongoing operational and maintenance costs,fraction of health information technology penetration, andproductivity gains. Financial benefits were divided into

See also:

Web-OnlyAppendix TablesConversion of figure and tables into slides

Annals of Internal Medicine Improving Patient Care

E-12 © 2006 American College of Physicians

Improving Patient Care is a special section within Annals supported in part by the U.S. Department of Health and Human Services (HHS) Agency for Healthcare Research and Quality(AHRQ). The opinions expressed in this article are those of the authors and do not represent the position or endorsement of AHRQ or HHS.

monetized benefits (that is, benefits expressed in dollarterms) and nonmonetized benefits (that is, benefits thatcould not be directly expressed in dollar terms but could beassigned dollar values).

Data Sources and Search StrategyWe performed 2 searches (in November 2003 and Jan-

uary 2004) of the English-language literature indexed inMEDLINE (1995 to January 2004) using a broad set ofterms to maximize sensitivity. (See the full list of searchterms and sequence of queries in the full evidence report atwww.ahrq.gov.) We also searched the Cochrane CentralRegister of Controlled Trials, the Cochrane Database ofAbstracts of Reviews of Effects, and the Periodical Ab-stracts Database; hand-searched personal libraries kept bycontent experts and project staff; and mined bibliographiesof articles and systematic reviews for citations. We askedcontent experts to identify unpublished literature. Finally,we asked content experts and peer reviewers to identifynewly published articles up to April 2005.

Study Selection and ClassificationTwo reviewers independently selected for detailed re-

view the following types of articles that addressed theworkings or implementation of a health technology system:systematic reviews, including meta-analyses; descriptive“qualitative” reports that focused on exploration of barri-ers; and quantitative reports. We classified quantitative re-ports as “hypothesis-testing” if the investigators compareddata between groups or across time periods and used sta-tistical tests to assess differences. We further categorizedhypothesis-testing studies (for example, randomized andnonrandomized, controlled trials, controlled before-and-af-ter studies) according to whether a concurrent comparisongroup was used. Hypothesis-testing studies without a con-current comparison group included those using simplepre–post, time-series, and historical control designs. Re-maining hypothesis-testing studies were classified as cross-sectional designs and other. We classified quantitative re-ports as a “predictive analysis” if they used methods such asstatistical modeling or expert panel estimates to predictwhat might happen with implementation of health infor-mation technology rather than what has happened. Thesestudies typically used hybrid methods—frequently mixingprimary data collection with secondary data collection plusexpert opinion and assumptions—to make quantitative es-timates for data that had otherwise not been empiricallymeasured. Cost-effectiveness and cost–benefit studies gen-erally fell into this group.

Data Extraction and SynthesisTwo reviewers independently appraised and extracted

details of selected articles using standardized abstractionforms and resolved discrepancies by consensus. We thenused narrative synthesis methods to integrate findings intodescriptive summaries. Each institution that accounted formore than 5% of the total sample of 257 papers was des-ignated as a benchmark research leader. We grouped syn-

theses by institution and by whether the systems were com-mercially or internally developed.

Role of the Funding SourcesThis work was produced under Agency for Healthcare

Research and Quality contract no. 2002. In addition to theAgency for Healthcare Research and Quality, this workwas also funded by the Office of the Assistant Secretary forPlanning and Evaluation, U.S. Department of Health andHuman Services, and the Office of Disease Prevention andHealth Promotion, U.S. Department of Health and Hu-man Services. The funding sources had no role in the de-sign, analysis, or interpretation of the study or in the deci-sion to submit the manuscript for publication.

DATA SYNTHESIS

Literature Selection OverviewOf 867 articles, we rejected 140 during initial screen-

ing: 124 for not having health information technology asthe subject, 3 for not reporting relevant outcomes, and 13for miscellaneous reasons (categories not mutually exclu-sive). Of the remaining 727 articles, we excluded the 470descriptive reports that did not examine barriers (Figure).We recorded details of and summarized each of the 257articles that we did include in an interactive database (http:

Key Summary Points

Health information technology has been shown to im-prove quality by increasing adherence to guidelines, en-hancing disease surveillance, and decreasing medicationerrors.

Much of the evidence on quality improvement relates toprimary and secondary preventive care.

The major efficiency benefit has been decreased utilizationof care.

Effect on time utilization is mixed.

Empirically measured cost data are limited and inconclu-sive.

Most of the high-quality literature regarding multifunc-tional health information technology systems comes from4 benchmark research institutions.

Little evidence is available on the effect of multifunctionalcommercially developed systems.

Little evidence is available on interoperability and con-sumer health information technology.

A major limitation of the literature is its generalizability.

Improving Patient CareImpact of Health Information Technology on Quality, Efficiency, and Costs

www.annals.org 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 E-13

//healthit.ahrq.gov/tools/rand) that serves as the evidencetable for our report (14). Twenty-four percent of all studiescame from the following 4 benchmark institutions: 1) theRegenstrief Institute, 2) Brigham and Women’s Hospital/Partners Health Care, 3) the Department of Veterans Af-fairs, and 4) LDS Hospital/ Intermountain Health Care.

Types and Functions of Technology SystemsThe reports addressed the following types of primary

systems: decision support aimed at providers (63%), elec-tronic health records (37%), and computerized providerorder entry (13%). Specific functional capabilities of sys-tems that were described in reports included electronicdocumentation (31%), order entry (22%), results manage-ment (19%), and administrative capabilities (18%). Only8% of the described systems had specific consumer healthcapabilities, and only 1% had capabilities that allowed sys-tems from different facilities to connect with each otherand share data interoperably. Most studies (n � 125) as-sessed the effect of the systems in the outpatient setting. Ofthe 213 hypothesis-testing studies, 83 contained some dataon costs.

Several studies assessed interventions with limitedfunctionality, such as stand-alone decision support systems(15–17). Such studies provide limited information aboutissues that today’s decision makers face when selecting andimplementing health information technology. Thus, wepreferentially highlight in the following paragraphs studiesthat were conducted in the United States, that had empir-ically measured data on multifunctional systems, and thatincluded health information and data storage in the formof electronic documentation or order-entry capabilities.Predictive analyses were excluded. Seventy-four studies metthese criteria: 52 from the 4 benchmark leaders and 22from other institutions.

Data from Benchmark InstitutionsThe health information technology systems evaluated

by the benchmark leaders shared many characteristics. Allthe systems were multifunctional and included decisionsupport, all were internally developed by research experts atthe respective academic institutions, and all had capabilitiesadded incrementally over several years. Furthermore, mostreported studies of these systems used research designs withhigh internal validity (for example, randomized, controlledtrials).

Appendix Table 1 (18–69) (available at www.annals.org) provides a structured summary of each study from the4 benchmark institutions. This table also includes studiesthat met inclusion criteria not highlighted in this synthesis(26, 27, 30, 39, 40, 53, 62, 65). The data supported 5primary themes (3 directly related to quality and 2 address-ing efficiency). Implementation of a multifunctional healthinformation technology system had the following effects:1) increased delivery of care in adherence to guidelines andprotocols, 2) enhanced capacity to perform surveillanceand monitoring for disease conditions and care delivery, 3)reductions in rates of medication errors, 4) decreased uti-lization of care, and 5) mixed effects on time utilization.

Effects on Quality

The major effect of health information technology onquality of care was its role in increasing adherence to guide-line- or protocol-based care. Decision support, usually inthe form of computerized reminders, was a component ofall adherence studies. The decision support functions wereusually embedded in electronic health records or comput-erized provider order-entry systems. Electronic healthrecords systems were more frequently examined in the out-patient setting; provider order-entry systems were more of-ten assessed in the inpatient setting. Improvements in pro-cesses of care delivery ranged from absolute increases of 5to 66 percentage points, with most increases clustering inthe range of 12 to 20 percentage points.

Twelve of the 20 adherence studies examined the ef-fects of health information technology on enhancing pre-ventive health care delivery (18, 21–25, 29, 31–33, 35, 37).Eight studies included measures for primary preventive

Table. Health Information Technology Frameworks*

Framework Basis (Reference) Elements

Componentsof an HITimplementation

Expert consensus Technological (e.g., system applica-tions)

Organizational process change (e.g.,workflow redesign)

Human factors (e.g., user-friendliness)Project management (e.g., achieving

project milestones)Types of HIT

systemsExpert consensus Electronic health records

Computerized provider order entryDecision support (stand-alone sys-

tems)Electronic results reporting (stand-

alone systems)Electronic prescribingConsumer health informatics/patient

decision supportMobile computingTelemedicine (data interchange–

based)Electronic health communicationAdministrationData exchange networksKnowledge retrieval systemsHIT in generalOther

Functionalcapabilitiesof an HITsystem†

Institute of Medi-cine’s “key ca-pabilities” ofan electronichealth record(13)

Clinical documentation (health infor-mation/data)

Results managementOrder entry managementDecision supportElectronic communication and con-

nectivityPatient supportAdministrative processesReporting and population health

* HIT � health information technology.† Assumes the electronic health record is the foundation for a comprehensive HITsystem.

Improving Patient Care Impact of Health Information Technology on Quality, Efficiency, and Costs

E-14 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 www.annals.org

care (18, 21–25, 31, 33), 4 studies included secondarypreventive measures (29, 33, 35, 37), and 1 study assessedscreening (not mutually exclusive) (32). The most com-mon primary preventive measures examined were rates ofinfluenza vaccination (improvement, 12 to 18 percentagepoints), pneumococcal vaccinations (improvement, 20 to33 percentage points), and fecal occult blood testing (im-provement, 12 to 33 percentage points) (18, 22, 24).

Three studies examined the effect of health informa-tion technology on secondary preventive care for compli-cations related to hospitalization. One clinical controlledtrial that used computerized surveillance and identificationof high-risk patients plus alerts to physicians demonstrateda 3.3–percentage point absolute decrease (from 8.2% to4.9%) in a combined primary end point of deep venous

thrombosis and pulmonary embolism in high-risk hospital-ized patients (29). One time-series study showed a 5–per-centage point absolute decrease in prevention of pressureulcers in hospitalized patients (35), and another showed a0.4–percentage point absolute decrease in postoperative in-fections (37).

While most evidence for health information technolo-gy–related quality improvement through enhanced adher-ence to guidelines focused on preventive care, other studiescovered a diverse range for types of care, including hyper-tension treatment (34), laboratory testing for hospitalizedpatients, and use of advance directives (see Appendix Table 1,available at www.annals.org, for the numeric effects) (19).

The second theme showed the capacity of health in-formation technology to improve quality of care through

Figure. Search flow for health information technology (HIT) literature.

Improving Patient CareImpact of Health Information Technology on Quality, Efficiency, and Costs

www.annals.org 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 E-15

clinical monitoring based on large-scale screening and ag-gregation of data. These studies demonstrated how healthinformation technology can support new ways of deliver-ing care that are not feasible with paper-based informationmanagement. In one study, investigators screened morethan 90 000 hospital admissions to identify the frequencyof adverse drug events (43); they found a rate of 2.4 events/100 admissions. Adverse drug events were associated withan absolute increase in crude mortality of 2.45 percentagepoints and an increase in costs of $2262, primarily due toa 1.9-day increase in length of stay. Two studies fromEvans and colleagues (44, 45) reported using an electronichealth record to identify adverse drug events, examine theircause, and develop programs to decrease their frequency.In the first study, the researchers designed interventions onthe basis of electronic health record surveillance that in-creased absolute adverse drug event identification by 2.36percentage points (from 0.04% to 2.4%) and decreasedabsolute adverse drug event rates by 5.4 percentage points(from 7.6% to 2.2%) (44). The report did not describedetails of the interventions used to reduce adverse drugevents. In the second study, the researchers used electronichealth record surveillance of nearly 61 000 inpatient ad-missions to determine that adverse drug events cause a1.9-day increase in length of hospital stay and an increaseof $1939 in charges (45).

Three studies from the Veterans Affairs system exam-ined the surveillance and data aggregation capacity ofhealth information technology systems for facilitating qual-ity-of-care measurement. Automated quality measurementwas found to be less labor intensive, but 2 of the studiesfound important methodologic limitations that affectedthe validity of automated quality measurement. For exam-ple, 1 study found high rates of false-positive results withuse of automated quality measurement and indicated thatsuch approaches may yield biased results (41). The secondstudy found that automated queries from computerizeddisease registries underestimated completion of quality-of-care processes when compared with manual chart abstrac-tion of electronic health records and paper chart sources(42).

Finally, 2 studies examined the role of health informa-tion technology surveillance systems in identifying infec-tious disease outbreaks. The first study found that use of acounty-based electronic system for reporting results led to a29–percentage point absolute increase in cases of shigello-sis identified during an outbreak and a 2.5-day decrease inidentification and public health reporting time (38). Thesecond study showed a 14–percentage point absolute in-crease in identification of hospital-acquired infections anda 65% relative decrease in identification time (from 130 to46 hours) (46).

The third health information technology–mediated ef-fect on quality was a reduction in medication errors. Twostudies of computerized provider order entry from LDSHospital (51, 52) showed statistically significant decreases

in adverse drug events, and a third study by Bates andcolleagues (49) showed a non–statistically significant trendtoward decreased drug events and a large decrease in med-ication errors. The first LDS Hospital study used a cohortwith historical control design to evaluate the effect of com-puterized alerts on antibiotic use (52). Compared with a2-year preintervention period, many statistically significantimprovements were noted, including a decrease in antibi-otic-associated adverse drug events (from 28 to 4 events),decreased length of stay (from 13 to 10 days), and a reduc-tion in total hospital costs (from $35 283 to $26 315). Thesecond study from LDS Hospital demonstrated a 0.6–per-centage point (from 0.9% to 0.3%) absolute decrease inantibiotic-associated adverse drug events (51).

Bates and colleagues examined adverse events andshowed a 17% non–statistically significant trend toward adecrease in these events (49). Although this outcome didnot reach statistical significance, adverse drug events werenot the main focus of the evaluation. The primary endpoint for this study was a surrogate end point for adversedrug events: nonintercepted serious medication errors.This end point demonstrated a statistically significant 55%relative decrease. The results from this trial were furthersupported by a second, follow-up study by the same re-searchers examining the long-term effect of the imple-mented system (48). After the first published study, theresearch team analyzed adverse drug events not preventedby computerized provider order entry, and the level ofdecision support was increased. This second study used atime-series design and found an 86% relative decrease innonintercepted serious medication errors.

Health information technology systems also decreasedmedication errors by improving medication dosing. Im-provements in dosing ranged from 12% to 21%; the pri-mary outcome examined was doses prescribed within therecommended range and centered on antibiotics and anti-coagulation (47, 50, 51).

Effects on Efficiency

Studies examined 2 primary types of technology-re-lated effects on efficiency: utilization of care and providertime. Ten studies examined the effect of health informa-tion technology systems on utilization of care. Eightshowed decreased rates of health services utilization (54–61); computerized provider order-entry systems that pro-vided decision support at the point of care were the pri-mary interventions leading to decreased utilization. Typesof decision support included automated calculation of pre-test probability for diagnostic tests, display of previous testresults, display of laboratory test costs, and computerizedreminders. Absolute decreases in utilization rates rangedfrom 8.5 to 24 percentage points. The primary servicesaffected were laboratory and radiology testing. Most stud-ies did not judge the appropriateness of the decrease inservice utilization but instead reported the effect of health

Improving Patient Care Impact of Health Information Technology on Quality, Efficiency, and Costs

E-16 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 www.annals.org

information technology on the level of utilization. Moststudies did not directly measure cost savings. Instead, re-searchers translated nonmonetized decreases in servicesinto monetized estimates through the average cost of theexamined service at that institution. One large study fromTierney and colleagues examined direct total costs per ad-mission as its main end point and found a 12.7% absolutedecrease (from $6964 to $6077) in costs associated with a0.9-day decrease in length of stay (57).

The effect of health information technology on pro-vider time was mixed. Two studies from the RegenstriefInstitute examining inpatient order entry showed increasesin physician time related to computer use (57, 64). An-other study on outpatient use of electronic health recordsfrom Partners Health Care showed a clinically negligibleincrease in clinic visit time of 0.5 minute (67). Studiessuggested that time requirements decreased as physiciansgrew used to the systems, but formal long-term evaluationswere not available. Two studies showed slight decreases indocumentation-related nursing time (68, 69) that were dueto the streamlining of workflow. One study examinedoverall time to delivery of care and found an 11% decreasein time to deliver treatment through the use of computer-ized order entry with alerts to physician pagers (66).

Effects on Costs

Data on costs were more limited than the evidence onquality and efficiency. Fifteen of the 52 studies containedsome data on costs (20, 28, 31, 36, 43, 47, 50–52, 54–58,63). Most of the cost data available from the institutionalleaders were related to changes in utilization of services dueto health information technology. Only 3 studies had costdata on aspects of system implementation or maintenance.Two studies provided computer storage costs; these weremore than 20 years old, however, and therefore were oflimited relevance (28, 58). The third reported that systemmaintenance costs were $700 000 (31). Because these sys-tems were built, implemented, and evaluated incrementallyover time, and in some cases were supported by researchgrants, it is unlikely that total development and implemen-tation costs could be calculated accurately and in full de-tail.

Data from Other Institutions about MultifunctionalSystems

Appendix Table 2 (available at www.annals.org) sum-marizes the 22 studies (70–91) from the other institutions.Most of these studies evaluated internally developed sys-tems in academic institutions. The types of benefits foundin these studies were similar to those demonstrated inbenchmark institutions, although an additional theme wasrelated to initial implementation costs. Unlike most studiesfrom the benchmark institutions, which used randomizedor controlled clinical trial designs, the most common de-signs of the studies from other institutions were pre–postand time-series designs that lacked a concurrent compari-

son group. Thirteen of the 22 studies evaluated internallydeveloped systems (70–82). Only 9 evaluated commercialhealth information technology systems. Because many de-cision makers are likely to consider implementing a com-mercially developed system rather than internally develop-ing their own, we detail these 9 studies in the followingparagraphs.

Two studies examined the effect of systems on utiliza-tion of care (83, 84). Both were set in Kaiser Permanente’sPacific Northwest region and evaluated the same electronichealth record system (Epic Systems Corp., Verona, Wis-consin) at different periods through time-series designs.One study (1994–1997) supported the findings of thebenchmark institutions, showing decreased utilization of 2radiology tests after implementation of electronic healthrecords (83), while the second study (2000–2004) showedno conclusive decreases in utilization of radiology and lab-oratory services (84). Unlike the reports from the bench-mark institutions, this second study also showed no statis-tically significant improvements in 3 process measures ofquality. It did find a statistically significant decrease inage-adjusted total office visits per member: a relative de-crease of 9% in year 2 after implementation of the elec-tronic health record. Telephone-based care showed a rela-tive increase of 65% over the same time. A third studyevaluated this electronic health record and focused on effi-ciency; it showed that physicians took 30 days to return totheir baseline level of productivity after implementationand that visit time increased on average by 2 minutes perencounter (85).

Two studies that were part of the same randomizedtrial from Rollman and colleagues, set at the University ofPittsburgh, examined the use of an electronic health record(MedicaLogic Corp., Beaverton, Oregon) with decisionsupport in improving care for depression (86, 87). The firststudy evaluated electronic health record–based monitoringto enhance depression screening. As in the monitoringstudies from the benchmark institutions, electronic healthrecord screening was found to support new ways of orga-nizing care. Physicians agreed with 65% of the computer-screened diagnoses 3 days after receiving notification of theresults. In the second phase of the trial, 2 different elec-tronic health record–based decision support interventionswere implemented to improve adherence to guideline-based care for depression. Unlike the effects on adherenceseen in the benchmark institutions, neither interventionshowed statistically significant differences when comparedwith usual care.

Two pre–post studies from Ohio State University eval-uated the effect of a commercial computerized order-entrysystem (Siemens Medical Solutions Health Services Corp.,Malvern, Pennsylvania) on time utilization and medicationerrors (88, 89). As in the benchmark institutions, time tocare dramatically decreased compared with the period be-fore the order-entry system was implemented. Relativedecreases in other outcomes were as follows: medication

Improving Patient CareImpact of Health Information Technology on Quality, Efficiency, and Costs

www.annals.org 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 E-17

turnaround time, 64% (88) and 73% (89); radiology com-pletion time, 43% (88) and 24% (89); and results report-ing time, 25% (88). Use of computerized provider orderentry had large effects on medication errors in both studies.Before implementation, 11.3% (88) and 13% (89) of or-ders had transcription errors; afterward, these errors wereentirely eliminated. One study assessed length of stay andfound that it decreased 5%; total cost of hospitalization,however, showed no statistically significant differences(88). In contrast, a third study examining the effect oforder entry on nurse documentation time showed no ben-efits (90).

In contrast to all previous studies on computer order-entry systems, a study by Koppel and colleagues used amixed quantitative–qualitative approach to investigate thepossible role of such a system (Eclipsys Corp., Boca Raton,Florida) in facilitating medication prescribing errors (91).Twenty-two types of medication error risks were found tobe facilitated by computer order entry, relating to 2 basiccauses: fragmentation of data and flaws in human–machineinterface.

These 9 studies infrequently reported or measureddata on costs and contextual factors. Two reported infor-mation on costs (88, 90). Neither described the total initialcosts of purchasing or implementing the system being eval-uated. Data on contextual factors such as reimbursementmix, degree of capitation, and barriers encountered duringimplementation were scant; only 2 studies included suchinformation. The study by Koppel and colleagues (91) in-cluded detailed contextual information related to humanfactors. One health record study reported physician class-room training time of 16 hours before implementation(85). Another order-entry study reported that nurses re-ceived 16 hours of training, clerical staff received 8 hours,and physicians received 2 to 4 hours (89).

DISCUSSION

To date, the health information technology literaturehas shown many important quality- and efficiency-relatedbenefits as well as limitations relating to generalizabilityand empirical data on costs. Studies from 4 benchmarkleaders demonstrate that implementing a multifunctionalsystem can yield real benefits in terms of increased deliveryof care based on guidelines (particularly in the domain ofpreventive health), enhanced monitoring and surveillanceactivities, reduction of medication errors, and decreasedrates of utilization for potentially redundant or inappropri-ate care. However, the method used by the benchmarkleaders to get to this point—the incremental developmentover many years of an internally designed system led byacademic research champions—is unlikely to be an optionfor most institutions contemplating implementation ofhealth information technology.

Studies from these 4 benchmark institutions havedemonstrated the efficacy of health information technology

for improving quality and efficiency. However, the effec-tiveness of these technologies in the practice settings wheremost health care is delivered remains less clear. Effective-ness and generalizability are of particular importance inthis field because health information technologies are toolsthat support the delivery of care—they do not, in and ofthemselves, alter states of disease or of health. As such, howthese tools are used and the context in which they areimplemented are critical (92–94).

For providers considering a commercially available sys-tem installed as a package, only a limited body of literatureis available to inform decision making. The available evi-dence comes mainly from time-series or pre–post studies,derives from a staff-model managed care organization oracademic health centers, and concerns a limited number ofprocess measures. These data, in general, support the find-ings of studies from the benchmark institutions on theeffect of health information technology in reducing utili-zation and medication errors. However, they do not sup-port the findings of increased adherence to protocol-basedcare. Published evidence of the information needed tomake informed decisions about acquiring and implement-ing health information technology in community settingsis nearly nonexistent. For example, potentially importantevidence related to initial capital costs, effect on providerproductivity, resources required for staff training (such astime and skills), and workflow redesign is difficult to locatein the peer-reviewed literature. Also lacking are key data onfinancial context, such as degree of capitation, which hasbeen suggested by a model to be an important factor indefining the business case for electronic health record use(95).

Several systematic reviews related to health informa-tion technology have been done. However, they have beenlimited to specific systems, such as computerized providerorder entry (96); capabilities, such as computerized re-minders (97, 98); or clinical specialty (99). No study todate has reviewed a broad range of health informationtechnologies. In addition, to make our findings as relevantas possible to the broad range of stakeholders interestedin health information technology, we developed a Web-hosted database of our research findings. This databaseallows different stakeholders to find the literature most rel-evant to their implementation circumstances and their in-formation needs.

This study has several important limitations. The firstrelates to the quantity and scope of the literature. Althoughwe did a comprehensive search, we identified only a limitedset of articles with quantitative data. In many importantdomains, we found few studies. This was particularly trueof health information technology applications relevant toconsumers and to interoperability, areas critical to the ca-pacity for health information technology to fundamentallychange health care. A second limitation relates to synthe-sizing the effect of a broad range of technologies. We at-tempted to address this limitation by basing our work on

Improving Patient Care Impact of Health Information Technology on Quality, Efficiency, and Costs

E-18 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 www.annals.org

well-defined analytic frameworks and by identifying notonly the systems used but also their functional capabilities.A third relates to the heterogeneity in reporting. Descrip-tions of health information technology systems were oftenvery limited, making it difficult to assess whether somesystem capabilities were absent or simply not reported.Similarly, limited information was reported on the overallimplementation process and organizational context.

This review raises many questions central to a broadrange of stakeholders in health care, including providers,consumers, policymakers, technology experts, and privatesector vendors. Adoption of health information technologyhas become one of the few widely supported, bipartisaninitiatives in the fragmented, often contentious health caresector (100). Currently, numerous pieces of state and fed-eral legislation under consideration seek to expand adop-tion of health information technology (101–103). Healthcare improvement organizations such as the LeapfrogGroup are strongly advocating adoption of health informa-tion technology as a key aspect of health care reform. Pol-icy discussions are addressing whether physician reimburse-ment should be altered, with higher reimbursements forthose who use health information technology (104). Twocritical questions that remain are 1) what will be the ben-efits of these initiatives and 2) who will pay and who willbenefit?

Regarding the former, a disproportionate amount ofliterature on the benefits that have been realized comesfrom a small set of early-adopter institutions that imple-mented internally developed health information technol-ogy systems. These institutions had considerable expertisein health information technology and implemented sys-tems over long periods in a gradual, iterative fashion. Miss-ing from this literature are data on how to implementmultifunctional health information technology systems inother health care settings. Internally developed systems areunlikely to be feasible as models for broad-scale use ofhealth information technology. Most practices and organi-zations will adopt a commercially developed health infor-mation technology system, and, given logistic constraintsand budgetary issues, their implementation cycles will bemuch shorter. The limited quantitative and qualitative de-scription of the implementation context significantly ham-pers how the literature on health information technologycan inform decision making by a broad array of stakehold-ers interested in this field.

With respect to the business case for health informa-tion technology, we found little information that couldempower stakeholders to judge for themselves the financialeffects of adoption. For instance, basic cost data needed todetermine the total cost of ownership of a system or of thereturn on investment are not available. Without these data,the costs of health information technology systems can beestimated only through complex predictive analysis andstatistical modeling methods, techniques generally notavailable outside of research. One of the chief barriers to

adoption of health information technology is the misalign-ment of incentives for its use (105, 106). Specifying poli-cies to address this barrier is hindered by the lack of costdata.

This review suggests several important future direc-tions in the field. First, additional studies need to evaluatecommercially developed systems in community settings,and additional funding for such work may be needed. Sec-ond, more information is needed regarding the organiza-tional change, workflow redesign, human factors, andproject management issues involved with realizing benefitsfrom health information technology. Third, a high prioritymust be the development of uniform standards for thereporting of research on implementation of health infor-mation technology, similar to the Consolidated Standardsof Reporting Trials (CONSORT) statements for random-ized, controlled trials and the Quality of Reporting ofMeta-analyses (QUORUM) statement for meta-analyses(107, 108). Finally, additional work is needed on interop-erability and consumer health technologies, such as thepersonal health record.

The advantages of health information technology overpaper records are readily discernible. However, withoutbetter information, stakeholders interested in promoting orconsidering adoption may not be able to determine whatbenefits to expect from health information technology use,how best to implement the system in order to maximizethe value derived from their investment, or how to directpolicy aimed at improving the quality and efficiency deliv-ered by the health care sector as a whole.

From the Southern California Evidence Based Practice Center, whichincludes RAND, Santa Monica, California; and University of California,Los Angeles, Cedars-Sinai Medical Center, and the Greater Los AngelesVeterans Affairs System, Los Angeles, California.

Disclaimer: The authors of this article are responsible for its contents.No statement in this article should be construed as an official position ofthe Agency for Healthcare Research and Quality. Statements made inthis publication do not represent the official policy or endorsement of theAgency or the U.S. government.

Acknowledgments: The authors thank the Veterans Affairs/Universityof California, Los Angeles, Robert Wood Johnson Clinical Scholars Pro-gram, the University of California, Los Angeles, Division of GeneralInternal Medicine and Health Services Research, and RAND for theirsupport during this research.

Grant Support: This work was produced under Agency for HealthcareResearch and Quality contract no. 2002. In addition to the Agency forHealthcare Research and Quality, this work was also funded by theOffice of the Assistant Secretary for Planning and Evaluation, U.S. De-partment of Health and Human Services, and the Office of DiseasePrevention and Health Promotion, U.S. Department of Health and Hu-man Services.

Potential Financial Conflicts of Interest: None disclosed.

Requests for Single Reprints: Basit Chaudhry, MD, Division of Gen-

Improving Patient CareImpact of Health Information Technology on Quality, Efficiency, and Costs

www.annals.org 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 E-19

eral Internal Medicine, University of California, Los Angeles, 911 Brox-ton Avenue, 2nd Floor, Los Angeles, CA 90095; e-mail,BChaudhry@mednet.ucla.edu.

Current author addresses are available at www.annals.org.

References1. Institute of Medicine. To Err Is Human: Building a Safer Health System.Washington, DC: National Academy Pr; 2000.2. Institute of Medicine. Crossing the Quality Chasm: A New Health System forthe 21st Century. Washington, DC: National Academy Pr; 2001.3. The Decade of Health Information Technology: Delivering Consumer-centricand Information-rich Health Care. A Strategic Framework. Bethesda, Maryland:Office of the National Coordinator for Health Information Technology, U.S.Department of Health and Human Services; 2004.4. Asch SM, McGlynn EA, Hogan MM, Hayward RA, Shekelle P, RubensteinL, et al. Comparison of quality of care for patients in the Veterans Health Ad-ministration and patients in a national sample. Ann Intern Med. 2004;141:938-45. [PMID: 15611491]5. Epstein AM, Lee TH, Hamel MB. Paying physicians for high-quality care. NEngl J Med. 2004;350:406-10. [PMID: 14736934]6. Smith MF. E-Health: Roadmap for 21st Century Health Care Consumers.Paris: Organisation for Economic Co-operation and Development Forum 2004:Health of Nations; 2004.7. Innovators and Visionaries: Strategies for Creating a Person-centered HealthSystem. FACCT: Foundation for Accountability; September 2003. Accessed atwww.markle.org/resources/facct/doclibFiles/documentFile_599.pdf on 13 March2006.8. Chassin MR, Galvin RW. The urgent need to improve health care quality.Institute of Medicine National Roundtable on Health Care Quality. JAMA.1998;280:1000-5. [PMID: 9749483]9. Ash JS, Gorman PN, Seshadri V, Hersh WR. Computerized physician orderentry in U.S. hospitals: results of a 2002 survey. J Am Med Inform Assoc. 2004;11:95-9. [PMID: 14633935]10. Ash JS, Stavri PZ, Kuperman GJ. A consensus statement on considerationsfor a successful CPOE implementation. J Am Med Inform Assoc. 2003;10:229-34. [PMID: 12626376]11. Valdes I, Kibbe DC, Tolleson G, Kunik ME, Petersen LA. Barriers toproliferation of electronic medical records. Inform Prim Care. 2004;12:3-9.[PMID: 15140347]12. Audet AM, Doty MM, Peugh J, Shamasdin J, Zapert K, Schoenbaum S.Information technologies: when will they make it into physicians’ black bags?MedGenMed. 2004;6:2. [PMID: 15775829]13. Key Capabilities of an Electronic Health Record System. Washington, DC:Institute of Medicine, Committee on Data Standards for Patient Safety Board onHealth Care Services; 2003.14. The Health Information Technology Interactive Database. Accessed at http://healthit.ahrq.gov/tools/rand on 17 March 2006.15. Goldman L, Cook EF, Brand DA, Lee TH, Rouan GW, Weisberg MC, etal. A computer protocol to predict myocardial infarction in emergency depart-ment patients with chest pain. N Engl J Med. 1988;318:797-803. [PMID:3280998]16. White RH, Hong R, Venook AP, Daschbach MM, Murray W, MungallDR, et al. Initiation of warfarin therapy: comparison of physician dosing withcomputer-assisted dosing. J Gen Intern Med. 1987;2:141-8. [PMID: 3295148]17. Burton ME, Ash CL, Hill DP Jr, Handy T, Shepherd MD, Vasko MR. Acontrolled trial of the cost benefit of computerized bayesian aminoglycoside ad-ministration. Clin Pharmacol Ther. 1991;49:685-94. [PMID: 1905602]18. Dexter PR, Perkins SM, Maharry KS, Jones K, McDonald CJ. Inpatientcomputer-based standing orders vs physician reminders to increase influenza andpneumococcal vaccination rates: a randomized trial. JAMA. 2004;292:2366-71.[PMID: 15547164]19. Dexter PR, Wolinsky FD, Gramelspacher GP, Zhou XH, Eckert GJ, Wais-burd M, et al. Effectiveness of computer-generated reminders for increasing dis-cussions about advance directives and completion of advance directive forms. Arandomized, controlled trial. Ann Intern Med. 1998;128:102-10. [PMID:9441569]20. Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomized trial

of “corollary orders” to prevent errors of omission. J Am Med Inform Assoc.1997;4:364-75. [PMID: 9292842]21. Overhage JM, Tierney WM, McDonald CJ. Computer reminders to imple-ment preventive care guidelines for hospitalized patients. Arch Intern Med. 1996;156:1551-6. [PMID: 8687263]22. Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring physiciansto respond to computerized reminders improves their compliance with preventivecare protocols. J Gen Intern Med. 1993;8:311-7. [PMID: 8320575]23. McDonald CJ, Hui SL, Tierney WM. Effects of computer reminders forinfluenza vaccination on morbidity during influenza epidemics. MD Comput.1992;9:304-12. [PMID: 1522792]24. Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician per-formance versus immediate reminders to perform preventive care. Effects onphysician compliance. Med Care. 1986;24:659-66. [PMID: 3736141]25. McDonald CJ, Hui SL, Smith DM, Tierney WM, Cohen SJ, WeinbergerM, et al. Reminders to physicians from an introspective computer medicalrecord. A two-year randomized trial. Ann Intern Med. 1984;100:130-8. [PMID:6691639]26. McDonald CJ, Wilson GA, McCabe GP Jr. Physician response to computerreminders. JAMA. 1980;244:1579-81. [PMID: 7420656]27. McDonald CJ. Protocol-based computer reminders, the quality of care andthe non-perfectability of man. N Engl J Med. 1976;295:1351-5. [PMID:988482]28. McDonald CJ. Use of a computer to detect and respond to clinical events: itseffect on clinician behavior. Ann Intern Med. 1976;84:162-7. [PMID: 1252043]29. Kucher N, Koo S, Quiroz R, Cooper JM, Paterno MD, Soukonnikov B, etal. Electronic alerts to prevent venous thromboembolism among hospitalizedpatients. N Engl J Med. 2005;352:969-77. [PMID: 15758007]30. Abookire SA, Teich JM, Sandige H, Paterno MD, Martin MT, KupermanGJ, et al. Improving allergy alerting in a computerized physician order entrysystem. Proc AMIA Symp. 2000:2-6. [PMID: 11080034]31. Teich JM, Merchia PR, Schmiz JL, Kuperman GJ, Spurr CD, Bates DW.Effects of computerized physician order entry on prescribing practices. Arch In-tern Med. 2000;160:2741-7. [PMID: 11025783]32. Cannon DS, Allen SN. A comparison of the effects of computer and manualreminders on compliance with a mental health clinical practice guideline. J AmMed Inform Assoc. 2000;7:196-203. [PMID: 10730603]33. Demakis JG, Beauchamp C, Cull WL, Denwood R, Eisen SA, Lofgren R,et al. Improving residents’ compliance with standards of ambulatory care: resultsfrom the VA Cooperative Study on Computerized Reminders. JAMA. 2000;284:1411-6. [PMID: 10989404]34. Rossi RA, Every NR. A computerized intervention to decrease the use ofcalcium channel blockers in hypertension. J Gen Intern Med. 1997;12:672-8.[PMID: 9383135]35. Willson D, Ashton C, Wingate N, Goff C, Horn S, Davies M, et al.Computerized support of pressure ulcer prevention and treatment protocols. ProcAnnu Symp Comput Appl Med Care. 1995:646-50. [PMID: 8563366]36. Evans RS, Classen DC, Pestotnik SL, Lundsgaarde HP, Burke JP. Improv-ing empiric antibiotic selection using computer decision support. Arch InternMed. 1994;154:878-84. [PMID: 8154950]37. Larsen RA, Evans RS, Burke JP, Pestotnik SL, Gardner RM, Classen DC.Improved perioperative antibiotic use and reduced surgical wound infectionsthrough use of computer decision analysis. Infect Control Hosp Epidemiol.1989;10:316-20. [PMID: 2745959]38. Overhage J, Suico J, McDonald C. Electronic laboratory reporting: barriers,solutions, and findings. J Public Health Manag Pract. 2001;7:60-6.39. Honigman B, Lee J, Rothschild J, Light P, Pulling RM, Yu T, et al. Usingcomputerized data to identify adverse drug events in outpatients. J Am MedInform Assoc. 2001;8:254-66. [PMID: 11320070]40. Jha AK, Kuperman GJ, Teich JM, Leape L, Shea B, Rittenberg E, et al.Identifying adverse drug events: development of a computer-based monitor andcomparison with chart review and stimulated voluntary report. J Am Med InformAssoc. 1998;5:305-14. [PMID: 9609500]41. Kramer TL, Owen RR, Cannon D, Sloan KL, Thrush CR, Williams DK,et al. How well do automated performance measures assess guideline implemen-tation for new-onset depression in the Veterans Health Administration? Jt CommJ Qual Saf. 2003;29:479-89. [PMID: 14513671]42. Kerr EA, Smith DM, Hogan MM, Krein SL, Pogach L, Hofer TP, et al.Comparing clinical automated, medical record, and hybrid data sources for dia-betes quality measures. Jt Comm J Qual Improv. 2002;28:555-65. [PMID:

Improving Patient Care Impact of Health Information Technology on Quality, Efficiency, and Costs

E-20 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 www.annals.org

12369158]43. Classen DC, Pestotnik SL, Evans RS, Lloyd JF, Burke JP. Adverse drugevents in hospitalized patients. Excess length of stay, extra costs, and attributablemortality. JAMA. 1997;277:301-6. [PMID: 9002492]44. Evans RS, Pestotnik SL, Classen DC, Bass SB, Burke JP. Prevention ofadverse drug events through computerized surveillance. Proc Annu Symp Com-put Appl Med Care. 1992:437-41. [PMID: 1482913]45. Evans RS, Classen DC, Stevens LE, Pestotnik SL, Gardner RM, Lloyd JF,et al. Using a hospital information system to assess the effects of adverse drugevents. Proc Annu Symp Comput Appl Med Care. 1993:161-5. [PMID:8130454]46. Evans RS, Larsen RA, Burke JP, Gardner RM, Meier FA, Jacobson JA, etal. Computer surveillance of hospital-acquired infections and antibiotic use.JAMA. 1986;256:1007-11. [PMID: 3735626]47. Chertow GM, Lee J, Kuperman GJ, Burdick E, Horsky J, Seger DL, et al.Guided medication dosing for inpatients with renal insufficiency. JAMA. 2001;286:2839-44. [PMID: 11735759]48. Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma’Luf N, et al. Theimpact of computerized physician order entry on medication error prevention. JAm Med Inform Assoc. 1999;6:313-21. [PMID: 10428004]49. Bates DW, Leape LL, Cullen DJ, Laird N, Petersen LA, Teich JM, et al.Effect of computerized physician order entry and a team intervention on preven-tion of serious medication errors. JAMA. 1998;280:1311-6. [PMID: 9794308]50. Mullett CJ, Evans RS, Christenson JC, Dean JM. Development and impactof a computerized pediatric antiinfective decision support program. Pediatrics.2001;108:E75. [PMID: 11581483]51. Evans RS, Pestotnik SL, Classen DC, Burke JP. Evaluation of a computer-assisted antibiotic-dose monitor. Ann Pharmacother. 1999;33:1026-31. [PMID:10534212]52. Evans RS, Pestotnik SL, Classen DC, Clemmer TP, Weaver LK, Orme JFJr, et al. A computer-assisted management program for antibiotics and otherantiinfective agents. N Engl J Med. 1998;338:232-8. [PMID: 9435330]53. White KS, Lindsay A, Pryor TA, Brown WF, Walsh K. Application of acomputerized medical decision-making process to the problem of digoxin intox-ication. J Am Coll Cardiol. 1984;4:571-6. [PMID: 6381570]54. Tierney WM, McDonald CJ, Hui SL, Martin DK. Computer predictionsof abnormal test results. Effects on outpatient testing. JAMA. 1988;259:1194-8.[PMID: 3339821]55. Tierney WM, McDonald CJ, Martin DK, Rogers MP. Computerized dis-play of past test results. Effect on outpatient testing. Ann Intern Med. 1987;107:569-74. [PMID: 3631792]56. Tierney WM, Miller ME, McDonald CJ. The effect on test ordering ofinforming physicians of the charges for outpatient diagnostic tests. N Engl J Med.1990;322:1499-504. [PMID: 2186274]57. Tierney WM, Miller ME, Overhage JM, McDonald CJ. Physician inpatientorder writing on microcomputer workstations. Effects on resource utilization.JAMA. 1993;269:379-83. [PMID: 8418345]58. Wilson GA, McDonald CJ, McCabe GP Jr. The effect of immediate accessto a computerized medical record on physician test ordering: a controlled clinicaltrial in the emergency room. Am J Public Health. 1982;72:698-702. [PMID:7046482]59. Chen P, Tanasijevic MJ, Schoenenberger RA, Fiskio J, Kuperman GJ,Bates DW. A computer-based intervention for improving the appropriateness ofantiepileptic drug level monitoring. Am J Clin Pathol. 2003;119:432-8. [PMID:12645347]60. Bates DW, Kuperman GJ, Rittenberg E, Teich JM, Fiskio J, Ma’luf N, etal. A randomized trial of a computer-based intervention to reduce utilization ofredundant laboratory tests. Am J Med. 1999;106:144-50. [PMID: 10230742]61. Shojania KG, Yokoe D, Platt R, Fiskio J, Ma’luf N, Bates DW. Reducingvancomycin use utilizing a computer guideline: results of a randomized controlledtrial. J Am Med Inform Assoc. 1998;5:554-62. [PMID: 9824802]62. Fihn SD, McDonell MB, Vermes D, Henikoff JG, Martin DC, CallahanCM, et al. A computerized intervention to improve timing of outpatient follow-up: a multicenter randomized trial in patients treated with warfarin. NationalConsortium of Anticoagulation Clinics. J Gen Intern Med. 1994;9:131-9.[PMID: 8195911]63. Steele MA, Bess DT, Franse VL, Graber SE. Cost effectiveness of twointerventions for reducing outpatient prescribing costs. DICP. 1989;23:497-500.[PMID: 2500784]64. Overhage JM, Perkins S, Tierney WM, McDonald CJ. Controlled trial of

direct physician order entry: effects on physicians’ time utilization in ambulatoryprimary care internal medicine practices. J Am Med Inform Assoc. 2001;8:361-71. [PMID: 11418543]65. Kuperman GJ, Teich JM, Bates DW, Hiltz FL, Hurley JM, Lee RY, et al.Detecting alerts, notifying the physician, and offering action items: a comprehen-sive alerting system. Proc AMIA Annu Fall Symp. 1996:704-8. [PMID:8947756]66. Kuperman GJ, Teich JM, Tanasijevic MJ, Ma’Luf N, Rittenberg E, Jha A,et al. Improving response to critical laboratory results with automation: results ofa randomized controlled trial. J Am Med Inform Assoc. 1999;6:512-22. [PMID:10579608]67. Pizziferri L, Kittler AF, Volk LA, Honour MM, Gupta S, Wang S, et al.Primary care physician time utilization before and after implementation of anelectronic health record: a time-motion study. J Biomed Inform. 2005;38:176-88. [PMID: 15896691]68. Wong DH, Gallegos Y, Weinger MB, Clack S, Slagle J, Anderson CT.Changes in intensive care unit nurse task activity after installation of a third-generation intensive care unit information system. Crit Care Med.2003;31:2488-94. [PMID: 14530756]69. Pierpont GL, Thilgen D. Effect of computerized charting on nursing activityin intensive care. Crit Care Med. 1995;23:1067-73. [PMID: 7774218]70. Khoury AT. Support of quality and business goals by an ambulatory auto-mated medical record system in Kaiser Permanente of Ohio. Eff Clin Pract.1998;1:73-82. [PMID: 10187226]71. Garr DR, Ornstein SM, Jenkins RG, Zemp LD. The effect of routine use ofcomputer-generated preventive reminders in a clinical practice. Am J Prev Med.1993;9:55-61. [PMID: 8439440]72. Ornstein SM, Garr DR, Jenkins RG, Musham C, Hamadeh G, LancasterC. Implementation and evaluation of a computer-based preventive services sys-tem. Fam Med. 1995;27:260-6. [PMID: 7797005]73. Schriger DL, Baraff LJ, Rogers WH, Cretin S. Implementation of clinicalguidelines using a computer charting system. Effect on the initial care of healthcare workers exposed to body fluids. JAMA. 1997;278:1585-90. [PMID:9370504]74. Schriger DL, Baraff LJ, Buller K, Shendrikar MA, Nagda S, Lin EJ, et al.Implementation of clinical guidelines via a computer charting system: effect onthe care of febrile children less than three years of age. J Am Med Inform Assoc.2000;7:186-95. [PMID: 10730602]75. Safran C, Rind DM, Davis RB, Ives D, Sands DZ, Currier J, et al. Guide-lines for management of HIV infection with computer-based patient’s record.Lancet. 1995;346:341-6. [PMID: 7623532]76. Day F, Hoang LP, Ouk S, Nagda S, Schriger DL. The impact of a guide-line-driven computer charting system on the emergency care of patients withacute low back pain. Proc Annu Symp Comput Appl Med Care. 1995:576-80.[PMID: 8563351]77. Simon GE, VonKorff M, Rutter C, Wagner E. Randomised trial of moni-toring, feedback, and management of care by telephone to improve treatment ofdepression in primary care. BMJ. 2000;320:550-4. [PMID: 10688563]78. Baird TK, Broekemeier RL, Anderson MW. Effectiveness of a computer-supported refill reminder system. Am J Hosp Pharm. 1984;41:2395-7. [PMID:6507445]79. Sanders DL, Miller RA. The effects on clinician ordering patterns of acomputerized decision support system for neuroradiology imaging studies. ProcAMIA Symp. 2001:583-7. [PMID: 11825254]80. Potts AL, Barr FE, Gregory DF, Wright L, Patel NR. Computerized phy-sician order entry and medication errors in a pediatric critical care unit. Pediatrics.2004;113:59-63. [PMID: 14702449]81. Wells BJ, Lobel KD, Dickerson LM. Using the electronic medical record toenhance the use of combination drugs. Am J Med Qual. 2003;18:147-9. [PMID:12934950]82. Khoury A. Finding value in EMRs (electronic medical records). HealthManag Technol. 1997;18:34, 36. [PMID: 10169803]83. Chin HL, Wallace P. Embedding guidelines into direct physician orderentry: simple methods, powerful results. Proc AMIA Symp. 1999:221-5. [PMID:10566353]84. Garrido T, Jamieson L, Zhou Y, Wiesenthal A, Liang L. Effect of electronichealth records in ambulatory care: retrospective, serial, cross sectional study. BMJ.2005;330:581. [PMID: 15760999]85. Krall MA. Acceptance and performance by clinicians using an ambulatoryelectronic medical record in an HMO. Proc Annu Symp Comput Appl Med

Improving Patient CareImpact of Health Information Technology on Quality, Efficiency, and Costs

www.annals.org 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 E-21

Care. 1995:708-11. [PMID: 8563380]86. Rollman BL, Hanusa BH, Gilbert T, Lowe HJ, Kapoor WN, SchulbergHC. The electronic medical record. A randomized trial of its impact on primarycare physicians’ initial management of major depression [corrected]. Arch InternMed. 2001;161:189-97. [PMID: 11176732]87. Rollman BL, Hanusa BH, Lowe HJ, Gilbert T, Kapoor WN, SchulbergHC. A randomized trial using computerized decision support to improve treat-ment of major depression in primary care. J Gen Intern Med. 2002;17:493-503.[PMID: 12133139]88. Mekhjian HS, Kumar RR, Kuehn L, Bentley TD, Teater P, Thomas A, etal. Immediate benefits realized following implementation of physician order entryat an academic medical center. J Am Med Inform Assoc. 2002;9:529-39. [PMID:12223505]89. Cordero L, Kuehn L, Kumar RR, Mekhjian HS. Impact of computerizedphysician order entry on clinical practice in a newborn intensive care unit. JPerinatol. 2004;24:88-93. [PMID: 14872207]90. Kilgore ML, Flint D, Pearce R. The varying impact of two clinical informa-tion systems in a cardiovascular intensive care unit. J Cardiovasc Manag. 1998;9:31-5. [PMID: 10178729]91. Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, et al.Role of computerized physician order entry systems in facilitating medicationerrors. JAMA. 2005;293:1197-203. [PMID: 15755942]92. Aarts J, Doorewaard H, Berg M. Understanding implementation: the case ofa computerized physician order entry system in a large Dutch university medicalcenter. J Am Med Inform Assoc. 2004;11:207-16. [PMID: 14764612]93. Berg M. Patient care information systems and health care work: a sociotech-nical approach. Int J Med Inform. 1999;55:87-101. [PMID: 10530825]94. Berg M, Langenberg C, vd Berg I, Kwakkernaat J. Considerations forsociotechnical design: experiences with an electronic patient record in a clinicalcontext. Int J Med Inform. 1998;52:243-51. [PMID: 9848421]95. Wang SJ, Middleton B, Prosser LA, Bardon CG, Spurr CD, Carchidi PJ,et al. A cost-benefit analysis of electronic medical records in primary care. Am JMed. 2003;114:397-403. [PMID: 12714130]96. Kaushal R, Shojania KG, Bates DW. Effects of computerized physicianorder entry and clinical decision support systems on medication safety: a system-

atic review. Arch Intern Med. 2003;163:1409-16. [PMID: 12824090]97. Bennett JW, Glasziou PP. Computerised reminders and feedback in medi-cation management: a systematic review of randomised controlled trials. Med JAust. 2003;178:217-22. [PMID: 12603185]98. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ,Beyene J, et al. Effects of computerized clinical decision support systems onpractitioner performance and patient outcomes: a systematic review. JAMA.2005;293:1223-38. [PMID: 15755945]99. Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: system-atic review of published articles on primary care computing during 1980-97.BMJ. 2001;322:279-82. [PMID: 11157532]100. Frist B, Clinton H. How to heal health care. Washington Post. 25 August2004:A17.101. S.1262: Health Technology to Enhance Quality Act. 2005.102. S.1418: A bill to enhance the adoption of a nationwide inter operable healthinformation technology system and to improve the quality and reduce the costs ofhealth care in the United States. 2005.103. S.1355: A bill to enhance the adoption of health information technologyand to improve the quality and reduce the costs of healthcare in the UnitedStates. 2005.104. Leapfrog Group. Purchasing Principles. Accessed at www.leapfroggroup.org/for_members/what_does_it_mean/purchasing_principals on 17 March 2006.105. Miller RH, Sim I. Physicians’ use of electronic medical records: barriers andsolutions. Health Aff (Millwood). 2004;23:116-26. [PMID: 15046136]106. Poon EG, Blumenthal D, Jaggi T, Honour MM, Bates DW, Kaushal R.Overcoming barriers to adopting and implementing computerized physician or-der entry systems in U.S. hospitals. Health Aff (Millwood). 2004;23:184-90.[PMID: 15318579]107. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al.Improving the quality of reporting of randomized controlled trials. The CON-SORT statement. JAMA. 1996;276:637-9. [PMID: 8773637]108. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improv-ing the quality of reports of meta-analyses of randomised controlled trials: theQUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354:1896-900. [PMID: 10584742]

Improving Patient Care Impact of Health Information Technology on Quality, Efficiency, and Costs

E-22 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 www.annals.org

Current Author Addresses: Dr. Chaudhry: Division of General InternalMedicine, University of California, Los Angeles, 911 Broxton Avenue,2nd Floor, Los Angeles, CA 90095.Dr. Wang: Cedars-Sinai Health System, 8700 Beverly Boulevard, LosAngeles, CA 90048.

Drs. Wu, Mojica, and Shekelle, Ms. Maglione, and Ms. Roth: RANDCorporation, 1776 Main Street, Santa Monica, CA 90401.Dr. Morton: RTI International, 3040 Cornwallis Road, Research Trian-gle Park, NC 27709.

www.annals.org 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 W-1

App

endi

xT

able

1.B

ench

mar

kLe

ader

sin

Hea

lth

Info

rmat

ion

Tech

nolo

gyR

esea

rch*

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

52)

Inst

itut

ion

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Key

Find

ing

Qua

lity

adhe

renc

e( n

�20

)D

exte

ret

al.,

2004

(18)

,R

CT

Reg

enst

rief

Inst

itute

1998

–199

9D

S/EH

RIn

patie

ntC

ompu

ter-

base

dst

andi

ngor

ders

vs.

com

pute

rized

phys

icia

nre

min

ders

Effe

ctiv

enes

sA

dher

ence

/su

rvei

llanc

e12

–per

cent

age

poin

tab

solu

tein

crea

se(f

rom

30%

to42

%)

inin

fluen

zava

ccin

atio

nsan

d20

–per

cent

age

poin

tab

solu

tein

crea

se(f

rom

31%

to51

%)

inpn

eum

ococ

calv

acci

natio

nsin

the

stan

ding

-ord

ers

grou

p;co

mpu

ter

iden

tifie

d50

%an

d22

%of

hosp

italiz

edpa

tient

sas

elig

ible

for

influ

enza

and

pneu

moc

occa

lva

ccin

atio

ns,

resp

ectiv

ely;

19%

and

7%of

patie

nts

scre

ened

byco

mpu

ter

asel

igib

lefo

rin

fluen

zaan

dpn

eum

ococ

calv

acci

nes

stat

edth

atth

eyha

dpr

evio

usly

been

vacc

inat

edan

ddi

dno

tre

quire

anot

her

vacc

inat

ion

(dat

afr

omou

tsid

efa

cilit

ies

not

pres

ent

inev

alua

ted

syst

em)

Dex

ter

etal

.,19

98(1

9),

RC

TR

egen

strie

fIn

stitu

teN

SD

S/EH

RO

utpa

tient

Com

pute

r-ge

nera

ted,

pape

r-ba

sed

rem

inde

rson

plan

ning

for

end-

of-l

ifeca

revs

.us

ualc

are

(no

rem

inde

r)

Effe

ctiv

enes

sA

dher

ence

20–p

erce

ntag

epo

int

abso

lute

incr

ease

(fro

m4%

to24

%)

inph

ysic

ians

who

disc

usse

dad

vanc

edi

rect

ives

;11

–per

cent

age

poin

tab

solu

tein

crea

se(f

rom

4%to

15%

)in

phys

icia

nsca

ring

for

patie

nts

who

com

plet

edad

vanc

edca

repl

ans

Ove

rhag

eet

al.,

1997

(20)

,R

CT

Reg

enst

rief

Inst

itute

1992

–199

3D

S/C

POE

Inpa

tient

Poin

t-of

-car

eco

mpu

teriz

edre

min

ders

onad

here

nce

togu

idel

ine-

base

dca

revs

.us

ualc

are

(CPO

Ew

ithou

tev

alua

ted

rem

inde

rs)

Effe

ctiv

enes

s/ef

ficie

ncy

Adh

eren

ce24

–per

cent

age

poin

tab

solu

tein

crea

se(f

rom

22%

to46

%)

inad

here

nce

togu

idel

ines

;ad

here

nce

incr

ease

dfo

rim

med

iate

,24

-h,

and

tota

lhos

pita

lst

ays;

little

incr

ease

betw

een

imm

edia

tean

d24

-had

here

nce;

33%

rela

tive

decr

ease

(fro

m15

6to

105)

innu

mbe

rof

phar

mac

ist

inte

rven

tions

;no

stat

istic

ally

sign

ifica

ntdi

ffer

ence

inco

sts

orle

ngth

ofst

ay

Con

tinue

don

follo

win

gpa

ge

W-2 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 www.annals.org

App

endi

xT

able

1—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

52)

Inst

itut

ion

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Key

Find

ing

Ove

rhag

eet

al.,

1996

(21)

,R

CT

Reg

enst

rief

Inst

itute

1992

–199

3D

S/EH

RIn

patie

ntC

ompu

ter-

gene

rate

dre

min

ders

onus

eof

prev

entiv

eca

rese

rvic

esvs

.us

ualc

are

Effe

ctiv

enes

sA

dher

ence

No

stat

istic

ally

sign

ifica

ntef

fect

dem

onst

rate

d;hi

ghad

here

nce

tore

min

ders

was

antic

ipat

edbu

tno

tde

mon

stra

ted,

and

nom

echa

nism

toca

ptur

ere

ason

sfo

rno

nadh

eren

cew

asin

corp

orat

edLi

tzel

man

etal

.,19

93(2

2),

RC

TR

egen

strie

fIn

stitu

te19

89D

S/EH

RO

utpa

tient

Com

pute

rized

rem

inde

rsof

prev

entiv

eca

re;

com

paris

onw

asbe

twee

nre

quiri

ngph

ysic

ians

toac

know

ledg

eth

ere

min

der

vs.

usin

gre

min

der

alon

e

Effe

ctiv

enes

sA

dher

ence

Ingr

oup

requ

iring

ackn

owle

dgm

ent,

12–p

erce

ntag

epo

int

abso

lute

incr

ease

(fro

m49

%to

61%

)in

feca

loc

cult

bloo

dte

stin

gan

d7–

perc

enta

gepo

int

abso

lute

incr

ease

(fro

m47

%to

54%

)in

mam

mog

raph

y;no

stat

istic

ally

sign

ifica

ntim

prov

emen

tin

Papa

nico

laou

scre

enin

gM

cDon

ald

etal

.,19

92(2

3),

RC

TR

egen

strie

fIn

stitu

te19

78–1

981

DS

Out

patie

ntC

ompu

ter-

gene

rate

d,pa

per-

base

dre

min

ders

onne

edfo

rin

fluen

zava

ccin

atio

nvs

.no

com

pute

r-ba

sed

rem

inde

rs

Effe

ctiv

enes

sA

dher

ence

12%

–18%

abso

lute

incr

ease

(15.

6%–2

9.5%

inye

ar3)

inin

fluen

zava

ccin

atio

nra

tes

Tier

ney

etal

.(2

4),

1986

,R

CT

Reg

enst

rief

Inst

itute

1983

–198

4D

S/da

tasu

mm

ary/

EHR

Out

patie

ntTh

ree

inte

rven

tions

onpr

even

tive

heal

th:

1)co

mpu

ter-

gene

rate

d,pa

per-

base

dre

min

ders

prov

ided

each

visi

t;2)

com

pute

r-ge

nera

ted,

pape

r-ba

sed

care

sum

mar

ies

gene

rate

dm

onth

ly;

and

3)us

ual

care

Effe

ctiv

enes

sA

dher

ence

App

roxi

mat

eab

solu

teef

fect

sof

com

pute

rized

rem

inde

rs:

33–p

erce

ntag

epo

int

abso

lute

incr

ease

(fro

m25

%to

58%

)in

feca

loc

cult

bloo

dte

stin

g,33

–per

cent

age

poin

tab

solu

tein

crea

se(f

rom

5%to

38%

)in

pneu

moc

occa

lvac

cina

tion,

16–p

erce

ntag

epo

int

abso

lute

incr

ease

(8%

to24

%)

insc

reen

ing

mam

mog

raph

y,an

d12

–per

cent

age

poin

tab

solu

tein

crea

se(f

rom

12%

to24

%)

inm

etro

nida

zole

for

tric

hom

onas

infe

ctio

ns;

9ot

her

prev

entiv

eca

repr

oces

ses

eval

uate

dsh

owed

nost

atis

tical

lyor

clin

ical

lysi

gnifi

cant

impr

ovem

ent

with

rem

inde

rs;

effe

cts

ofre

min

ders

wer

egr

eate

rth

anth

ose

ofth

esu

mm

ary

repo

rts

McD

onal

det

al.,

1984

(25)

,C

CT

Reg

enst

rief

Inst

itute

1978

–198

0D

S/EH

RO

utpa

tient

Com

pute

r-ge

nera

ted,

pape

r-ba

sed

rem

inde

rson

adhe

renc

eto

prot

ocol

-bas

edca

revs

.us

ualc

are

(no

rem

inde

rs)

Effe

ctiv

enes

s/ef

ficie

ncy

Adh

eren

ce15

%–2

0%in

crea

sein

adhe

renc

eto

prot

ocol

-bas

edca

re;

grea

test

incr

ease

sse

enin

prev

entiv

eca

re(r

elat

ive

incr

ease

inut

iliza

tion,

200%

–400

%);

info

rmat

ion

othe

rth

anla

bora

tory

and

phar

mac

yda

taen

tere

dby

rese

arch

assi

stan

ts;

com

pute

rsy

stem

did

not

capt

ure

all

patie

ntda

ta

Con

tinue

don

follo

win

gpa

ge

www.annals.org 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 W-3

App

endi

xT

able

1—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

52)

Inst

itut

ion

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Key

Find

ing

McD

onal

det

al.,

1980

(26)

,C

CT

Reg

enst

rief

Inst

itute

NS

DS/

EHR

Out

patie

ntC

ompu

ter-

gene

rate

d,pa

per-

base

dre

min

ders

with

and

with

out

liter

atur

eci

tatio

nson

adhe

renc

eto

prot

ocol

-bas

edca

revs

.us

ualc

are

Effe

ctiv

enes

sA

dher

ence

/m

edic

aler

rors

19–p

erce

ntag

epo

int

abso

lute

incr

ease

(fro

m19

.8%

to38

.4%

)in

adhe

renc

eto

prot

ocol

-bas

edca

re;

min

imal

lear

ning

effe

cts

wer

ese

enw

hen

the

com

pute

rized

rem

inde

rsw

ere

turn

edof

f

McD

onal

d,19

76(2

7),

RC

TR

egen

strie

fIn

stitu

teN

SD

S/EH

RO

utpa

tient

Com

pute

r-ge

nera

ted,

pape

r-ba

sed

rem

inde

rson

adhe

renc

eto

prot

ocol

-bas

edca

refo

rdi

abet

es

Effe

ctiv

enes

s/sa

fety

/ef

ficie

ncy

Adh

eren

ce/

med

icat

ion

erro

rs

15–p

erce

ntag

epo

int

incr

ease

(fro

m11

%to

36%

and

from

13%

to28

%)

inad

here

nce

topr

otoc

ols;

com

pute

rsy

stem

sco

uld

not

capt

ure

all

labo

rato

ryda

take

ptin

pape

rch

art;

phys

icia

nsag

reed

with

am

axim

umof

57%

ofco

mpu

ter

reco

mm

enda

tions

;m

ost

rem

inde

rsre

late

dto

med

icat

ion

care

;co

st$2

/vis

itto

mai

ntai

nco

mpu

ter

reco

rdM

cDon

ald,

1976

(28)

,R

CT

Reg

enst

rief

Inst

itute

1975

DS/

EHR

Out

patie

ntC

ompu

ter-

gene

rate

d,pa

per-

base

dre

min

ders

onad

here

nce

topr

otoc

ol-b

ased

care

Effe

ctiv

enes

s/sa

fety

Adh

eren

ce/

med

icat

ion

erro

rs

29–p

erce

ntag

epo

int

abso

lute

incr

ease

(fro

m22

%to

51%

)in

adhe

renc

eto

prot

ocol

s;co

mpu

ter

syst

emco

uld

not

capt

ure

alll

abor

ator

yda

take

ptin

pape

rch

art;

mos

tre

min

ders

rela

ted

tom

edic

atio

n-ba

sed

care

Kuc

her

etal

.,20

05(2

9),

CC

TPa

rtne

rsH

ealth

Car

e20

00–2

004

DS/

CPO

EIn

patie

ntC

ompu

teriz

edal

erts

for

antic

oagu

latio

npr

ophy

laxi

svs

.us

ualc

are

for

prev

entio

nof

veno

usth

rom

boem

bolis

min

high

-ris

kho

spita

lized

patie

nts

Effe

ctiv

enes

sA

dher

ence

/su

rvei

llanc

e3.

3–pe

rcen

tage

poin

tab

solu

tede

crea

se(f

rom

8.2%

to4.

9%)

inth

eco

mbi

ned

prim

ary

end

poin

tof

deep

veno

usth

rom

bosi

sor

pulm

onar

yem

bolis

mw

ithin

90d

afte

rho

spita

lizat

ion;

19–p

erce

ntag

epo

int

abso

lute

incr

ease

(fro

m33

.5%

to14

.5%

)in

use

ofan

ticoa

gula

tion

prop

hyla

xis;

aspa

rtof

inte

rven

tion,

aco

mpu

ter-

base

dris

kas

sess

men

tpr

ogra

mw

asus

edto

scre

enan

did

entif

yin

patie

nts

athi

ghris

kfo

rve

nous

thro

mbo

embo

lism

;80

%of

patie

nts

enro

lled

had

som

ety

peof

canc

erA

book

ireet

al.,

2000

(30)

,tim

e-se

ries

stud

y

Part

ners

Hea

lthC

are

1995

–199

9D

S/C

POE

Inpa

tient

Tren

dsin

aler

tsan

dph

ysic

ian

resp

onse

toal

erts

Safe

ty/

effe

ctiv

enes

sA

dher

ence

24–p

erce

ntag

epo

int

abso

lute

redu

ctio

n(f

rom

51%

to27

%)

inad

here

nce

to�d

efin

ite�

med

icat

ion

alle

rgy

aler

tsan

d26

–per

cent

age

poin

tab

solu

tere

duct

ion

inad

here

nce

(fro

m46

%to

20%

)to

�pos

sibl

e�al

lerg

yal

erts

over

4y;

adhe

renc

ede

crea

sed

asnu

mbe

rof

aler

tsin

crea

sed

Con

tinue

don

follo

win

gpa

ge

W-4 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 www.annals.org

App

endi

xT

able

1—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

52)

Inst

itut

ion

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Key

Find

ing

Teic

het

al.,

2000

(31)

,pr

e–po

stst

udy

Brig

ham

and

Wom

en’s

Hos

pita

l/Pa

rtne

rsH

ealth

Car

e

1993

CPO

E/D

SIn

patie

ntEf

fect

ofC

POE

onph

ysic

ian

pres

crib

ing

prac

tices

and

adhe

renc

eto

med

icat

ion

form

ular

ies

Effe

ctiv

enes

s/sa

fety

Adh

eren

ce66

–per

cent

age

poin

tab

solu

tein

crea

se(f

rom

15.6

%to

81.3

%)

info

rmul

ary

adhe

renc

efo

rga

stric

H2-b

lock

ers;

23–p

erce

ntag

epo

int

abso

lute

incr

ease

(fro

m24

%to

47%

)in

appr

opria

teus

eof

subc

utan

eous

hepa

rinpr

ophy

laxi

s;1.

5–pe

rcen

tage

poin

tab

solu

tede

crea

se(f

rom

2.1%

to0.

6%)

innu

mbe

rof

med

icat

ion

dose

sw

ritte

nth

atex

ceed

edm

axim

umre

com

men

ded

(�po

ssib

lybe

caus

eof

incr

ease

dus

eof

orde

rse

ts�)

;ef

fect

spe

rsis

ted

at1-

and

2-y

follo

w-u

p;co

stsa

ving

sfr

omH

2-b

lock

er,

$250

000;

cost

sto

mai

ntai

nsy

stem

,$7

0000

0/y

Can

non

and

Alle

n20

00(3

2),

RC

TV

A19

98D

S/EH

RO

utpa

tient

Gui

delin

e-ba

sed

com

pute

rized

vs.

man

ual

pape

r-ba

sed

rem

inde

rson

scre

enin

gra

tes

for

moo

ddi

sord

ers

Effe

ctiv

enes

sA

dher

ence

25.5

–per

cent

age

poin

tab

solu

tein

crea

se(f

rom

61%

to86

.5%

)in

phys

icia

nsc

reen

ing

for

moo

ddi

sord

ers

with

com

pute

rized

syst

em

Dem

akis

etal

.,20

00(3

3),

RC

TV

A19

95–1

996

DS/

EHR

Out

patie

ntC

ompu

teriz

edre

min

ders

onph

ysic

ian

adhe

renc

eto

ambu

lato

ryca

rere

com

men

datio

nsvs

.us

ualc

are

(EH

Rw

ithou

thy

pert

ensi

onre

min

ders

)

Effe

ctiv

enes

sA

dher

ence

5.3–

perc

enta

gepo

int

abso

lute

incr

ease

(fro

m53

.5%

to58

.8%

)in

adhe

renc

eto

reco

mm

ende

dca

re;

5of

13ex

amin

edca

repr

oces

ses

impr

oved

;ef

fect

ofre

min

ders

decr

ease

dov

ertim

eR

ossi

and

Ever

y,19

97(3

4),

RC

TV

A19

96D

SO

utpa

tient

Com

pute

r-ge

nera

ted,

pape

r-ba

sed

rem

inde

rson

adhe

renc

eto

appr

opria

teca

refo

rhy

pert

ensi

ontr

eatm

ent

vs.

usua

lcar

e(n

ore

min

ders

)

Effe

ctiv

enes

sA

dher

ence

11.3

–per

cent

age

poin

tab

solu

tein

crea

se(f

rom

�1%

to11

.3%

)in

appr

opria

tehy

pert

ensi

ontr

eatm

ent

Will

son

etal

.,19

95(3

5),

pre–

post

stud

y

LDS

Hos

pita

l/In

term

ount

ain

Hea

lthC

are

1994

–NS

DS

Inpa

tient

Com

pute

rized

guid

elin

esfo

rpr

even

tion

and

trea

tmen

tof

pres

sure

ulce

rs

Effe

ctiv

enes

sA

dher

ence

5–pe

rcen

tage

poin

tab

solu

tede

crea

se(f

rom

7%to

2%)

inul

cer

deve

lopm

ent Con

tinue

don

follo

win

gpa

ge

www.annals.org 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 W-5

App

endi

xT

able

1—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

52)

Inst

itut

ion

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Key

Find

ing

Evan

set

al.,

1994

(36)

,R

CT

LDS

Hos

pita

l/In

term

ount

ain

Hea

lthC

are

1990

DS/

EHR

Inpa

tient

Com

pute

rized

guid

elin

eson

appr

opria

tene

ssof

antib

iotic

use

Effe

ctiv

enes

sA

dher

ence

Com

pute

rpr

ogra

msu

gges

ted

corr

ect

antib

iotic

in94

%of

case

s;17

–per

cent

age

poin

tab

solu

tein

crea

se(f

rom

77%

to94

%)

inco

vera

geof

iden

tifie

dor

gani

sm;

27%

rela

tive

decr

ease

(fro

m22

to16

h)in

time

toap

prop

riate

trea

tmen

taf

ter

cultu

rere

sults

;21

%re

lativ

ede

crea

se(f

rom

$51.

93to

$41.

08)

inan

tibio

ticco

st;

phys

icia

nsor

dere

dap

prop

riate

antib

iotic

sw

ithin

12h

ofcu

lture

colle

ctio

nsi

gnifi

cant

lym

ore

ofte

nw

ithus

eof

prog

ram

com

pare

dw

ithus

ualc

are;

88%

ofph

ysic

ians

wou

ldre

com

men

dth

epr

ogra

mto

othe

rph

ysic

ians

,85

%sa

idth

epr

ogra

mim

prov

edan

tibio

ticse

lect

ion,

and

81%

said

use

impr

oved

care

Lars

enet

al.,

1989

(37)

,pr

e–po

stst

udy

LDS

Hos

pita

l/In

term

ount

ain

Hea

lthC

are

1985

–198

6D

SIn

patie

ntC

ompu

teriz

edre

min

ders

onap

prop

riate

ness

ofpr

eope

rativ

ean

tibio

tics

and

onra

tes

ofpo

stop

erat

ive

wou

ndin

fect

ions

Effe

ctiv

enes

sA

dher

ence

0.4–

perc

enta

gepo

int

abso

lute

decr

ease

(fro

m1.

1%to

0.7%

)in

tota

lpo

stop

erat

ive

wou

ndin

fect

ions

;0.

9–pe

rcen

tage

poin

tab

solu

tede

crea

se(f

rom

1.8%

to0.

9%)

inw

ound

infe

ctio

nsam

ong

patie

nts

with

gene

rali

ndic

atio

nfo

ran

tibio

ticpr

ophy

laxi

s;18

–per

cent

age

poin

tin

crea

se(f

rom

40%

to58

%)

inap

prop

riate

ness

ofan

tibio

tictim

ing

Surv

eilla

nce

(n�

9)O

verh

age

etal

.,20

01(3

8),

case

–con

trol

stud

y

Reg

enst

rief

Inst

itute

2000

–200

1El

ectr

onic

resu

ltsre

port

ing

Out

patie

ntEl

ectr

onic

labo

rato

ryre

port

ing

onpu

blic

heal

thsu

rvei

llanc

e

Acc

ess/

effe

ctiv

enes

sSu

rvei

llanc

e29

–per

cent

age

poin

tin

crea

se(f

rom

71%

to10

0%)

inid

entif

ied

case

sdu

ring

ash

igel

losi

sou

tbre

ak;

2.5-

dde

crea

sein

repo

rtin

gtim

eH

onig

man

etal

.,20

01(3

9),

coho

rtst

udy

Brig

ham

and

Wom

en’s

Hos

pita

l/Pa

rtne

rsH

ealth

Car

e

1995

–199

6EH

RO

utpa

tient

Com

pute

rpr

ogra

mto

retr

ospe

ctiv

ely

dete

ctA

DEs

vs.

char

tre

view

Safe

tySu

rvei

llanc

eSe

nsiti

vity

for

AD

Es,

58%

;sp

ecifi

city

,88

%;

AD

Era

tew

as5.

5/10

0pa

tient

s;9%

ofou

tpat

ient

AD

Esre

quire

dho

spita

lizat

ion

Jha

etal

.,19

98(4

0),

case

serie

sBr

igha

man

dW

omen

’sH

ospi

tal/

Part

ners

Hea

lthC

are

1995

Dat

a sum

mar

y/C

POE

Inpa

tient

Thre

ein

terv

entio

nsfo

rid

entif

ying

adve

rse

drug

even

ts:

1)co

mpu

ter

mon

itorin

g,2)

char

tre

view

,an

d3)

volu

ntar

yre

port

ing

Safe

tySu

rvei

llanc

eC

ompu

teriz

edm

onito

ring

iden

tifie

d45

%of

AD

Es;

char

tre

view

iden

tifie

d65

%;

volu

ntar

yre

port

ing

iden

tifie

d4%

;co

mpu

ter

was

bett

erfo

rA

DEs

rela

ted

toqu

antit

ativ

ech

ange

s(e

.g.,

labo

rato

ryva

lues

)an

dch

art

revi

eww

asbe

tter

for

AD

Esre

late

don

lyto

sym

ptom

s;vo

lunt

ary

repo

rtin

gw

asbe

tter

for

pote

ntia

lAD

Esth

atha

dno

tye

toc

curr

ed

Con

tinue

don

follo

win

gpa

ge

W-6 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 www.annals.org

App

endi

xT

able

1—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

52)

Inst

itut

ion

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Key

Find

ing

Kra

mer

etal

.,20

03(4

1),

case

serie

s

VA

1999

–200

0El

ectr

onic

data

colle

ctio

n/EH

R

Out

patie

ntA

utom

ated

data

colle

ctio

nal

gorit

hms

vs.

man

ual

revi

ewof

EHR

sby

trai

ned

abst

ract

ers

ondi

agno

sing

new

case

sof

depr

essi

on

Effe

ctiv

enes

sSu

rvei

llanc

eH

igh

fals

e-po

sitiv

era

tefo

rdi

agno

sis

via

auto

mat

edal

gorit

hms;

qual

ityin

dica

tor

scor

esba

sed

sole

lyon

auto

mat

edda

tash

owag

reem

ent

with

man

ualr

evie

w,

but

resu

ltsm

aysh

owso

me

bias

Ker

ret

al.,

2002

(42)

,ca

sese

ries

VA

1999

–200

0El

ectr

onic

data

colle

ctio

n/EH

R

Mix

edA

utom

ated

quer

ies

ofco

mpu

teriz

eddi

seas

ere

gist

ries

vs.

man

ualc

hart

abst

ract

ion

(mix

edEH

Ran

dpa

per

char

tso

urce

s)on

mea

surin

gqu

ality

ofca

re

Effe

ctiv

enes

sSu

rvei

llanc

eA

utom

ated

quer

ies

from

dise

ase

regi

strie

sun

dere

stim

ated

rate

sof

com

plet

ion

for

qual

ity-o

f-ca

repr

oces

sin

dica

tors

;no

diff

eren

ces

wer

eno

ted

for

inte

rmed

iate

outc

ome

mea

sure

s;au

tom

ated

quer

ies

wer

ele

ssla

bor-

inte

nsiv

eC

lass

enet

al.,

1997

(43)

,ca

se–

cont

rols

tudy

LDS

Hos

pita

l/In

term

ount

ain

Hea

lthC

are

1990

–199

3D

ata sum

mar

y/D

S/EH

R

Inpa

tient

Com

pute

rsu

rvei

llanc

eto

iden

tify

AD

Esan

das

soci

ated

cost

s

Effic

ienc

ySu

rvei

llanc

eC

ompu

ter

syst

emw

asus

edto

scre

en91

574

adm

issi

ons

for

AD

Es;

2.43

AD

Es/1

00ad

mis

sion

s;2.

45–a

bsol

ute

perc

enta

gepo

int

incr

ease

(fro

m1.

05%

to3.

5%)

incr

ude

mor

talit

yas

soci

ated

with

AD

Es;

1.9-

din

crea

sein

attr

ibut

able

leng

thof

stay

and

asso

ciat

ed$2

262

incr

ease

inco

sts

Evan

set

al.,

1992

(44)

,ca

se–c

ontr

olst

udy

LDS

Hos

pita

l/In

term

ount

ain

Hea

lthC

are

1989

–199

2D

ata sum

mar

y/EH

R

Inpa

tient

Com

pute

rized

surv

eilla

nce

vs.

man

ualr

epor

ting

onid

entif

ying

and

prev

entin

gA

DEs

Safe

tySu

rvei

llanc

e2.

36–p

erce

ntag

epo

int

abso

lute

incr

ease

(fro

m0.

04%

to2.

4%)

inid

entif

ied

AD

Es;

aler

ting

syst

emim

plem

ente

din

year

2in

whi

chph

arm

acis

tsre

ceiv

edsu

rvei

llanc

ere

port

san

dco

ntac

ted

phys

icia

ns;

seve

reA

DEs

decr

ease

d5.

4pe

rcen

tage

poin

ts(f

rom

7.6%

to2.

2%);

AD

Esdu

eto

alle

rgie

sde

crea

sed

13.6

perc

enta

gepo

ints

(fro

m15

%to

1.4%

);an

alys

isof

AD

Eda

taba

seal

low

edau

thor

sto

desi

gnre

duct

ion

initi

ativ

esth

atde

crea

sed

sign

ifica

ntA

DEs

from

56ev

ents

inye

ar1

to8

even

tsin

year

3Ev

ans

etal

.,19

93(4

5),

case

–con

trol

LDS

Hos

pita

l/In

term

ount

ain

Hea

lthC

are

1990

–199

2D

ata sum

mar

y/D

S/EH

R

Inpa

tient

Com

pute

rized

surv

eilla

nce

onde

term

inin

gat

trib

utab

leef

fect

ofA

DEs

onho

spita

llen

gth

ofst

ayan

dco

sts

Safe

ty/

effic

ienc

ySu

rvei

llanc

e/m

edic

atio

ner

rors

Com

pute

rsy

stem

was

used

tosc

reen

the

reco

rds

of60

836

inpa

tient

s;13

48A

DEs

wer

eid

entif

ied

in12

09pa

tient

s;th

ose

with

AD

Esw

ere

mat

ched

toa

tota

lof

1054

2co

ntro

lpa

tient

s;A

DEs

wer

eas

soci

ated

with

anat

trib

utab

lein

crea

sein

leng

thof

stay

of1.

9d

and

incr

ease

inat

trib

utab

leco

sts

of$1

939

Con

tinue

don

follo

win

gpa

ge

www.annals.org 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 W-7

App

endi

xT

able

1—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

52)

Inst

itut

ion

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Key

Find

ing

Evan

set

al.,

1986

(46)

,co

hort

stud

y

LDS

Hos

pita

l/In

term

ount

ain

Hea

lthC

are

1984

Dat

a sum

mar

y/EH

R

Inpa

tient

Com

pute

rvs

.m

anua

lsu

rvei

llanc

efo

ras

sess

ing

rate

sof

hosp

ital-

acqu

ired

infe

ctio

nsan

das

soci

ated

antib

iotic

use

Safe

tySu

rvei

llanc

e14

–per

cent

age

poin

tab

solu

tein

crea

se(f

rom

76%

to90

%)

inid

entif

icat

ion

ofin

fect

ions

;65

%de

crea

se(f

rom

130

to46

h)in

time

requ

ired

for

surv

eilla

nce;

4–pe

rcen

tage

poin

tab

solu

tein

crea

se(f

rom

19%

to23

%)

infa

lse-

posi

tive

rate

sw

ithth

eco

mpu

ter;

com

pute

rsc

reen

ing

iden

tifie

dpa

tient

sre

ceiv

ing

antib

iotic

sto

whi

chin

fect

ions

wer

ere

sist

ant,

antib

iotic

sw

ithle

ssex

pens

ive

alte

rnat

ives

,an

dpa

tient

sre

ceiv

ing

prop

hyla

ctic

antib

iotic

sfo

rlo

nger

than

nece

ssar

y

Med

icat

ion

erro

rs(n

�7)

Che

rtow

etal

.,20

01(4

7),

CC

TBr

igha

man

dW

omen

’sH

ospi

tal/

Part

ners

Hea

lthC

are

1997

–199

8D

S/C

POE

Inpa

tient

Com

pute

rized

drug

dosi

ngal

gorit

hmto

dete

rmin

eef

fect

onm

edic

atio

npr

escr

ibin

gin

rena

lin

suff

icie

ncy

vs.

usua

lcar

e(C

POE

with

out

algo

rithm

)

Effic

ienc

y/ef

fect

iven

ess

Med

icat

ion

erro

rs/

adhe

renc

e

21–p

erce

ntag

epo

int

abso

lute

incr

ease

(fro

m30

%to

51%

)in

appr

opria

tem

edic

atio

nor

ders

(dos

ing

leve

lsor

dosi

ngfr

eque

ncy)

;4.

5%re

duct

ion

(fro

m4.

5to

4.3

d)in

leng

thof

stay

;no

stat

istic

ally

sign

ifica

ntde

crea

sein

cost

sBa

tes

etal

.,19

99(4

8),

time-

serie

sst

udy

Brig

ham

and

Wom

en’s

Hos

pita

l/Pa

rtne

rsH

ealth

Car

e

1992

–199

7C

POE/

DS

Inpa

tient

CPO

Ew

ithen

hanc

edD

Son

rate

sof

nonm

isse

ddo

seer

rors

and

over

all

noni

nter

rupt

edse

rious

med

icat

ion

erro

rra

tes

Safe

tyM

edic

atio

ner

rors

86%

rela

tive

redu

ctio

n(f

rom

7.6

even

ts/1

000

patie

nt-d

ays

to1.

1ev

ents

/100

0pa

tient

-day

s)in

noni

nter

cept

edse

rious

med

icat

ion

erro

rs;

82%

rela

tive

redu

ctio

n(f

rom

142

even

ts/1

000

patie

nt-d

ays

to26

.6ev

ents

/100

0pa

tient

-day

s)in

nonm

isse

ddo

seer

rors

;re

duct

ions

seen

for

alle

rror

type

s;le

velo

fD

Sin

syst

emin

crea

sed

over

time

Bate

set

al.,

1998

(49)

,tim

e-se

ries

stud

y

Brig

ham

and

Wom

en’s

Hos

pita

l/Pa

rtne

rsH

ealth

Car

e

1993

–199

5C

POE/

DS

Inpa

tient

CPO

Eon

rate

sof

med

icat

ion

erro

rsan

dpr

even

tabl

eA

DEs

vs.

CPO

Ew

ithad

ditio

nof

team

chan

ges

Safe

tyM

edic

atio

ner

rors

55%

rela

tive

risk

redu

ctio

n(f

rom

10.7

even

ts/1

000

patie

nt-d

ays

to4.

9ev

ents

/100

0pa

tient

-day

s)in

noni

nter

cept

edse

rious

med

icat

ion

erro

rs;

non–

stat

istic

ally

sign

ifica

nt17

%re

lativ

ere

duct

ion

(fro

m4.

69/1

000

patie

nt-d

ays

to3.

86/1

000

patie

nt-d

ays)

inpr

even

tabl

eA

DEs

;de

crea

ses

seen

for

alll

evel

sof

erro

rse

verit

y;te

amch

ange

sco

nfer

red

noad

ditio

nalb

enef

itov

erC

POE

Con

tinue

don

follo

win

gpa

ge

W-8 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 www.annals.org

App

endi

xT

able

1—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

52)

Inst

itut

ion

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Key

Find

ing

Mul

lett

etal

.,20

01(5

0),

pre–

post

stud

y

LDS

Hos

pita

l/In

term

ount

ain

Hea

lthC

are

1998

–199

9D

S/EH

RPe

diat

ricIC

UC

ompu

teriz

edgu

idel

ines

onan

tibio

ticap

prop

riate

ness

and

use

Effe

ctiv

enes

s/ef

ficie

ncy

Med

icat

ion

erro

rs/

adhe

renc

e

32%

rela

tive

decr

ease

(fro

m15

.8to

10.8

)in

num

ber

ofda

ysth

atan

tibio

tics

wer

epr

escr

ibed

outs

ide

the

reco

mm

ende

ddo

sing

rang

e;59

%re

lativ

ede

crea

sein

aco

mpo

site

mea

sure

ofne

edfo

rph

arm

acis

tin

terv

entio

nsfo

rin

corr

ect

dosi

ng;

6.3–

perc

enta

gepo

int

abso

lute

incr

ease

(fro

m60

.2%

to66

.5%

)in

prop

ortio

nof

ICU

patie

nts

rece

ivin

gan

tibio

tics;

nost

atis

tical

lysi

gnifi

cant

diff

eren

ces

inov

eral

lant

ibio

ticco

sts;

wei

ghte

dan

tibio

tic-c

ost

stat

istic

show

edde

crea

sein

cost

sEv

ans

etal

.,19

99(5

1),

pre–

post

stud

y

LDS

Hos

pita

l/In

term

ount

ain

Hea

lthC

are

1993

–199

6D

S/EH

RIn

patie

ntC

ompu

teriz

edm

onito

ring

ofan

tibio

ticdo

ses

onap

prop

riate

ness

ofdo

sing

and

AD

Era

tes

Safe

tyM

edic

atio

ner

rors

/ut

iliza

tion

ofca

re

0.6–

perc

enta

gepo

int

abso

lute

decr

ease

(fro

m0.

9%to

0.3%

)in

antib

iotic

-ass

ocia

ted

AD

Es;

6%re

lativ

ede

crea

se(f

rom

50%

to40

%)

inpa

tient

sre

ceiv

ing

exce

ssan

tibio

ticdo

ses

for

�1

d;12

%re

lativ

ede

crea

se(f

rom

10.1

to8.

9do

ses)

innu

mbe

rof

antib

iotic

dose

spr

escr

ibed

and

13%

rela

tive

decr

ease

inco

st(f

rom

$92.

96to

$80.

62);

exce

ssdo

sing

was

asso

ciat

edw

ithin

crea

sed

AD

Era

tes

Evan

set

al.,

1998

(52)

,co

hort

stud

yw

ithhi

stor

ical

cont

rol

LDS

Hos

pita

l/In

term

ount

ain

Hea

lthC

are

1992

–199

5D

S/EH

RIn

patie

nt(I

CU

)C

ompu

teriz

edal

erts

onan

tibio

ticus

eEf

fect

iven

ess/

effic

ienc

y/sa

fety

Med

icat

ion

erro

rs/

adhe

renc

e

Com

pare

dw

ithth

e2-

ypr

eint

erve

ntio

npe

riod,

redu

ctio

nsw

ere

seen

for

the

follo

win

g:an

tibio

tic-a

ssoc

iate

dA

DEs

(28

vs.

4),

mis

mat

ches

betw

een

infe

ctio

nsu

scep

tibili

tyan

dan

tibio

tic(2

06vs

.12

epis

odes

),or

dere

ddr

ugs

for

whi

cha

patie

ntha

dan

alle

rgy

(146

vs.

35ep

isod

es),

days

ofex

cess

dosi

ng(f

rom

5.9

to2.

7d)

,an

tibio

ticco

sts

(fro

m$3

40to

$102

),le

ngth

ofst

ay(f

rom

13to

10d)

,an

dto

tal

hosp

italc

osts

(fro

m$3

528

3to

$26

315)

Whi

teet

al.,

1984

(53)

,R

CT

LDS

Hos

pita

l/In

term

ount

ain

Hea

lthC

are

NS

DS

Inpa

tient

Com

pute

r-ge

nera

ted,

pape

r-ba

sed

aler

tsy

stem

ondi

goxi

nto

xici

ty

Safe

tyM

edic

atio

ner

rors

2.8-

fold

incr

ease

inw

ithho

ldin

gdi

goxi

non

day

aler

tw

assi

gnal

ed;

2.7-

fold

incr

ease

inte

stin

gof

seru

mdi

goxi

nle

vels

inre

spon

seto

aler

ts;

over

all,

22%

incr

ease

inph

ysic

ian

actio

nsin

resp

onse

todi

goxi

n-re

late

dev

ents

(unw

eigh

ted

even

tra

tes

inst

udy

grou

psno

tpr

ovid

ed)

Con

tinue

don

follo

win

gpa

ge

www.annals.org 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 W-9

App

endi

xT

able

1—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

52)

Inst

itut

ion

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Key

Find

ing

Effi

cien

cy:

utili

zati

onof

care

(n�

10)

Tier

ney

etal

.,19

88(5

4),

RC

TR

egen

strie

f19

84–1

985

DS/

CPO

EO

utpa

tient

Com

pute

rpr

ogra

mth

atge

nera

tes

and

disp

lays

pret

est

prob

abili

ties

for

diag

nost

icte

sts

onut

iliza

tion

ofca

re

Effic

ienc

yU

tiliz

atio

nof

care

8.8%

decr

ease

(fro

m$1

2.27

to$1

1.18

)in

diag

nost

icte

stco

sts

per

patie

ntvi

sit;

grea

test

decr

ease

was

due

tore

duce

dut

iliza

tion

ofco

mpl

ete

bloo

dco

unts

and

elec

trol

ytes

(the

2m

ost

com

mon

test

s);

mea

nre

ceiv

er-

oper

atin

gch

arac

teris

ticcu

rve

for

com

pute

rpr

edic

tions

,0.

80Ti

erne

yet

al.,

1987

(55)

,pr

e–po

stst

udy

Reg

enst

rief

NS

DS/

CPO

E/EH

RO

utpa

tient

Com

pute

rpr

ogra

msh

owin

gpr

evio

uste

stre

sults

asph

ysic

ians

are

orde

ring

new

test

svs

.us

ualc

are

(no

disp

lay

ofpr

evio

uste

stre

sults

)

Effic

ienc

yU

tiliz

atio

nof

care

8.5%

decr

ease

(fro

m0.

56to

0.51

)in

num

ber

ofte

sts

orde

red

per

visi

t;13

%de

crea

se(f

rom

$13.

99to

$12.

17)

inte

stco

sts

per

visi

t;pr

ogra

mto

ok4

sto

disp

lay

past

resu

ltson

scre

enpe

rte

stTi

erne

yet

al.,

1990

(56)

,R

CT

Reg

enst

rief

1988

DS/

CPO

E/EH

RO

utpa

tient

Effe

ctof

info

rmat

ion

onpo

int-

of-c

are

test

cost

son

utili

zatio

nvs

.us

ualc

are

(CPO

Ew

ithou

tin

form

atio

non

test

cost

s)

Effic

ienc

yU

tiliz

atio

nof

care

14.3

%de

crea

se(f

rom

1.82

to1.

56)

innu

mbe

rof

diag

nost

icte

sts

orde

red

per

visi

t;12

.9%

decr

ease

(fro

m$5

1.81

to$4

5.13

)in

diag

nost

icte

stco

sts

per

visi

t;ef

fect

was

grea

test

for

sche

dule

dpa

tient

visi

tsTi

erne

yet

al.,

1993

(57)

,R

CT

Reg

enst

rief

1990

–199

1C

POE/

EHR

Inpa

tient

CPO

Eon

cost

san

dut

iliza

tion

ofhe

alth

care

Effic

ienc

yU

tiliz

atio

nof

care

12.7

%re

duct

ion

(fro

m$6

964

to$6

077)

into

talc

osts

per

adm

issi

on;

stat

istic

ally

sign

ifica

ntde

crea

ses

inho

spita

lbed

,m

edic

atio

n,an

ddi

agno

stic

test

cost

s;0.

9-d

decr

ease

(fro

m8.

5to

7.6

d)in

leng

thof

stay

;33

-min

incr

ease

inph

ysic

ian

time

spen

tor

derin

gte

sts

Wils

onet

al.,

1982

(58)

,R

CT

Reg

enst

rief

NS

EHR

Emer

genc

yde

part

men

tC

ompu

ter-

gene

rate

d,pa

per-

base

dpa

tient

reco

rdsu

mm

arie

son

utili

zatio

nof

care

vs.

usua

lca

re(n

oca

resu

mm

arie

s)

Effic

ienc

yU

tiliz

atio

nof

care

Stud

yin

terr

upte

dby

anin

terv

alpr

ogra

mm

ing

erro

rth

atpr

even

ted

the

prev

ious

4–6

mo

ofa

patie

nt’s

data

from

bein

gpr

inte

d,th

usdi

vidi

ngth

est

udy

into

2tim

epe

riods

,T1

and

T2;

18%

decr

ease

(fro

m3.

28te

sts/

visi

tto

2.23

test

s/vi

sit)

inte

sts

orde

red

for

med

ical

visi

tsdu

ring

T1;

14%

decr

ease

(fro

m$3

4.91

to$2

9.94

)in

cost

s;no

n–st

atis

tical

lysi

gnifi

cant

decr

ease

inte

sts

orde

red

for

surg

ical

visi

tsdu

ring

T1;

nost

atis

tical

lysi

gnifi

cant

diff

eren

cein

utili

zatio

nor

cost

sdu

ring

T2C

hen

etal

.,20

03(5

9),

pre–

post

stud

y

Brig

ham

and

Wom

en’s

Hos

pita

l/Pa

rtne

rsH

ealth

Car

e

1995

–199

9D

S/C

POE

Inpa

tient

Com

pute

rized

rem

inde

rson

rate

sof

inap

prop

riate

daily

test

ing

ofan

tiepi

lept

icdr

ugle

vels

Effe

ctiv

enes

s/ef

ficie

ncy

Util

izat

ion

ofca

re27

%de

crea

se(5

3of

200

tota

l)in

redu

ndan

tla

bora

tory

test

sof

antie

pile

ptic

med

icat

ion

leve

ls;

effe

ctof

rem

inde

rsst

able

over

4y

Con

tinue

don

follo

win

gpa

ge

W-10 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 www.annals.org

App

endi

xT

able

1—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

52)

Inst

itut

ion

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Key

Find

ing

Bate

set

al.,

1999

(60)

,R

CT

Brig

ham

and

Wom

en’s

Hos

pita

l/Pa

rtne

rsH

ealth

Car

e

1994

DS/

CPO

EIn

patie

ntC

ompu

teriz

edre

min

ders

onus

eof

labo

rato

ryte

sts,

redu

ndan

tor

derin

g,an

das

soci

ated

cost

svs

.us

ual

care

(CPO

Ew

ithou

tla

bora

tory

-rel

ated

rem

inde

rs)

Effic

ienc

yU

tiliz

atio

nof

care

24–p

erce

ntag

epo

int

abso

lute

redu

ctio

n(f

rom

51%

to27

%)

inre

dund

ant

test

s;31

%of

rem

inde

rsw

ere

over

ridde

nby

phys

icia

ns;

41%

ofov

errid

esw

ere

just

ified

;56

%of

redu

ndan

tte

sts

orde

red

inin

terv

entio

ngr

oup

wer

eno

tor

dere

dvi

aco

mpu

ter;

only

51%

ofre

dund

ant

test

sor

dere

din

cont

rolg

roup

wer

eac

tual

lype

rfor

med

;es

timat

edco

stsa

ving

s,$3

500

0/y

(0.1

5%of

tota

lla

bora

tory

expe

nditu

res)

Shoj

ania

etal

.,19

98(6

1),

RC

TBr

igha

man

dW

omen

’sH

ospi

tal/

Part

ners

Hea

lthC

are

1996

–199

7D

S/C

POE

Inpa

tient

Poin

t-of

-car

eco

mpu

teriz

edgu

idel

ines

onan

tibio

ticus

evs

.us

ualc

are

(CPO

Ew

ithou

tan

tibio

ticD

S)

Effe

ctiv

enes

sU

tiliz

atio

nof

care

32%

rela

tive

decr

ease

(fro

m16

.7or

ders

/phy

sici

anto

11.3

orde

rs/p

hysi

cian

)in

antib

iotic

orde

rs;

both

initi

alan

dre

new

alor

der

rate

sde

crea

sed

Fihn

etal

.,19

94(6

2),

RC

TV

AN

SA

dmin

istr

ativ

e/D

SO

utpa

tient

Com

pute

rized

sche

dulin

gsy

stem

onfo

llow

-up

time

for

antic

oagu

latio

nm

onito

ring

vs.

usua

lcar

e

Safe

ty/

acce

ss/

effe

ctiv

enes

s

Util

izat

ion

ofca

re/

med

icat

ion

erro

rs

App

roxi

mat

e6-

din

crea

se(f

rom

25to

31d)

info

llow

-up

appo

intm

ent

inte

rval

;no

stat

istic

ally

sign

ifica

ntdi

ffer

ence

sin

antic

oagu

latio

nle

vels

orco

mpl

icat

ion

rate

sSt

eele

etal

.,19

89(6

3),

RC

TV

A19

87–1

988

Dat

a sum

mar

y/D

SO

utpa

tient

Com

pute

r-ge

nera

ted,

pape

r-ba

sed

feed

back

ofm

edic

atio

nco

sts

vs.

in-p

erso

nph

arm

acis

tco

unse

ling

onph

ysic

ian

pres

crib

ing

cost

s

Effic

ienc

yU

tiliz

atio

nof

care

No

stat

istic

ally

sign

ifica

ntdi

ffer

ence

sin

cost

sw

ere

foun

dw

ithco

mpu

ter-

base

dsy

stem

vs.

phar

mac

ist

coun

selin

g

Effi

cien

cy:

tim

e(n

�6)

Ove

rhag

eet

al.,

2001

(64)

,R

CT,

time-

mot

ion

stud

y

Reg

enst

rief

1996

–199

8C

POE/

EHR

Out

patie

ntC

POE

onph

ysic

ian

time

utili

zatio

nEf

ficie

ncy

Tim

eut

iliza

tion

6.2%

incr

ease

(fro

m34

.2to

36.3

min

)in

phys

icia

ntim

epe

rcl

inic

visi

t;ph

ysic

ians

cont

inue

dto

use

pape

rde

spite

CPO

E,th

ereb

ydu

plic

atin

gta

sks;

with

expe

rienc

eth

ere

was

ano

n–st

atis

tical

lysi

gnifi

cant

decr

ease

inph

ysic

ian

time

ofap

prox

imat

ely

10%

(3.7

min

)pe

rcl

inic

visi

tK

uper

man

etal

.,19

96(6

5),

cros

s-se

ctio

nal

anal

ysis

Brig

ham

and

Wom

en’s

Hos

pita

l/Pa

rtne

rsH

ealth

Car

e

1995

–199

6D

SIn

patie

ntC

ompu

teriz

edal

erts

sent

via

page

ron

phys

icia

nre

spon

seto

serio

uscl

inic

alev

ents

Safe

ty/

effe

ctiv

enes

sTi

me

tout

iliza

tion/

resp

onse

Phys

icia

nsre

spon

ded

to70

%of

aler

ts(1

214

of17

30al

erts

)an

dim

med

iate

lypl

aced

orde

rsin

resp

onse

to39

%of

answ

ered

aler

ts;

phys

icia

nsre

spon

ded

to82

.5%

ofan

swer

edal

erts

(100

2of

1214

)in

�15

min

Con

tinue

don

follo

win

gpa

ge

www.annals.org 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 W-11

App

endi

xT

able

1—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

52)

Inst

itut

ion

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Key

Find

ing

Kup

erm

anet

al.,

1999

(66)

,R

CT

Brig

ham

and

Wom

en’s

Hos

pita

l/Pa

rtne

rsH

ealth

Car

e

1994

–199

5D

SIn

patie

ntC

ompu

teriz

edal

erts

sent

via

page

ron

phys

icia

nre

spon

setim

eto

criti

cal

labo

rato

ryre

sults

Safe

ty/

effe

ctiv

enes

sTi

me

toca

re/s

afet

y38

%de

crea

se(f

rom

1.6

to1.

0h)

inm

edia

ntim

eun

tiltr

eatm

ent

orde

red;

11%

decr

ease

inm

ean

time

(fro

m4.

6to

4.1

h)un

tiltr

eatm

ent;

nost

atis

tical

lysi

gnifi

cant

decr

ease

inA

DEs

Pizz

iferr

iet

al.,

2005

(67)

,pr

e–po

st,

time-

mot

ion

stud

y

Brig

ham

and

Wom

en’s

Hos

pita

l/Pa

rtne

rsH

ealth

Car

e

2001

–200

3EH

RO

utpa

tient

EHR

use

onph

ysic

ian

time

utili

zatio

nin

clin

icEf

ficie

ncy

Tim

eut

iliza

tion

0.5-

min

decr

ease

(fro

m27

.55

to27

.05

min

)in

clin

icvi

sit

time;

phys

icia

nsfe

ltEH

Rs

impr

oved

qual

ity,

acce

ss,

and

com

mun

icat

ion

but

nega

tivel

yaf

fect

edw

orkl

oad

Won

get

al.,

2003

(68)

,pr

e–po

st,

time-

mot

ion

stud

y

VA

NS

EHR

Inpa

tient

(IC

U)

Com

pute

rized

docu

men

tatio

non

nurs

ing

time

utili

zatio

n

Effic

ienc

yTi

me

utili

zatio

n10

.9–p

erce

ntag

epo

int

abso

lute

decr

ease

(fro

m35

.1%

to24

.2%

)in

docu

men

tatio

ntim

e;8.

8–pe

rcen

tage

poin

tab

solu

tein

crea

se(f

rom

31.3

%to

40.1

%)

intim

esp

ent

ondi

rect

patie

ntca

rePi

erpo

ntan

dTh

ilgen

,19

95(6

9),

pre–

post

stud

y

VA

NS

EHR

Inpa

tient

(IC

U)

Effe

ctof

com

pute

rized

nurs

ing

docu

men

tatio

non

ICU

nurs

es’

time

utili

zatio

nan

dw

orkf

low

Effic

ienc

yTi

me

utili

zatio

n7–

perc

enta

gepo

int

abso

lute

decr

ease

(fro

m17

%to

10%

)in

char

ting

time;

3–pe

rcen

tage

poin

tde

crea

se(f

rom

7%to

4%)

inda

ta-g

athe

ring

time;

10%

oftim

ew

assp

ent

atco

mpu

ter

revi

ewin

gda

ta;

noch

ange

intim

esp

ent

inpa

tient

s’ro

oms

*A

DE

�ad

vers

edr

ugev

ent;

CC

T�

cont

rolle

dcl

inic

altr

ial;

CPO

E�

com

pute

rize

dpr

ovid

eror

der

entr

y;D

S�

deci

sion

supp

ort;

EH

R�

elec

tron

iche

alth

reco

rd;

H2-b

lock

ers

�hi

stam

ine-

2–bl

ocke

rs;

HIT

�he

alth

info

rmat

ion

tech

nolo

gy;

ICU

�in

tens

ive

care

unit

;N

S�

not

spec

ified

;R

CT

�ra

ndom

ized

,co

ntro

lled

tria

l;V

A�

Dep

artm

ent

ofV

eter

ans

Aff

airs

.

W-12 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 www.annals.org

App

endi

xT

able

2.O

ther

Inst

itut

ions

Res

earc

hing

Hea

lth

Info

rmat

ion

Tech

nolo

gy*

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

22)

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Type

ofIn

stit

utio

nK

eyFi

ndin

gs

Com

mer

cial

lyde

velo

ped

syst

ems

Adh

eren

ceR

ollm

anet

al.,

2002

(87)

,R

CT

1997

–199

8D

S/EH

R(L

ogic

ian,

Med

icaL

ogic

Cor

p.)

Out

patie

ntTh

ree

inte

rven

tions

onsc

reen

ing

for

moo

ddi

sord

ers

and

phys

i-ci

anag

reem

ent

with

com

put-

er-b

ased

diag

nosi

s:1)

com

-pu

ter-

gene

rate

d,pa

per-

base

dre

min

ders

for

depr

essi

onw

ithtr

eatm

ent

reco

mm

enda

tions

;2)

com

pute

r-ge

nera

ted,

pa-

per-

base

dre

min

ders

with

di-

agno

sis

info

rmat

ion;

onqu

al-

ityof

depr

essi

onca

re,

and

3)us

ualc

are

Effe

ctiv

enes

sA

dher

ence

Aca

dem

icN

ost

atis

tical

lysi

gnifi

cant

diff

eren

cein

depr

essi

onsy

mpt

omsc

ores

orde

liver

yof

reco

mm

ende

dpr

o-ce

sses

ofde

pres

sion

care

for

ei-

ther

inte

rven

tion

grou

pw

hen

com

pare

dw

ithus

ualc

are

Uti

lizat

ion

ofca

reG

arrid

oet

al.,

2005

(84)

,re

tros

pect

ive

time-

serie

sst

udy

2000

–200

4EH

R(E

piC

are,

Epic

Syst

ems

Cor

p.)

Out

patie

ntEH

Ron

adhe

renc

eto

reco

m-

men

ded

care

and

effic

ienc

ym

easu

res

Effe

ctiv

enes

sA

dher

ence

Kai

ser

Perm

anen

teN

ost

atis

tical

lysi

gnifi

cant

diff

eren

cein

depr

essi

onsy

mpt

omsc

ores

orde

liver

yof

reco

mm

ende

dpr

o-ce

sses

ofde

pres

sion

care

for

ei-

ther

inte

rven

tion

grou

pw

hen

com

pare

dw

ithus

ualc

are

Chi

nan

dW

al-

lace

,19

99(8

3),

time-

serie

sst

udy

1994

–199

7EH

R/D

S(E

piC

are,

Epic

Syst

ems

Cor

p.)

Out

patie

ntEH

Rw

ithC

POE

and

DS

onad

-he

renc

eto

guid

elin

e-ba

sed

care

for

radi

olog

yse

rvic

esan

dm

edic

atio

nus

e

Qua

lity/

effe

ctiv

enes

sU

tiliz

atio

nof

care

Kai

ser

Perm

anen

te48

%re

lativ

ede

crea

se(f

rom

10.6

test

s/10

00to

5.6

test

s/10

00)

for

uppe

rga

stro

inte

stin

altr

act

radi

-ol

ogy

stud

ies

byye

ar4

afte

rEH

Rim

plem

enta

tion,

with

a33

–per

-ce

ntag

epo

int

abso

lute

incr

ease

(fro

m55

%to

88%

)in

adhe

r-en

ceto

prot

ocol

sfo

rte

stor

der-

ing;

20%

decr

ease

inch

est

ra-

diog

raph

sor

dere

d(y

ears

afte

rim

plem

enta

tion

and

info

rmat

ion

onre

lativ

eor

abso

lute

decr

ease

not

prov

ided

);2.

3–pe

rcen

tage

poin

tab

solu

tede

crea

se(f

rom

4.7%

to2.

4%)

inpr

escr

ibin

gof

ano

nfor

mul

ary

antid

epre

ssan

tby

year

2af

ter

impl

emen

tatio

n

Con

tinue

don

follo

win

gpa

ge

www.annals.org 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 W-13

App

endi

xT

able

2—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

22)

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Type

ofIn

stit

utio

nK

eyFi

ndin

gs

Surv

eilla

nce

Rol

lman

etal

.,20

01(8

6),

RC

T19

97–1

999

DS/

EHR

(Log

icia

n,M

edic

aLog

icC

orp.

)O

utpa

tient

Thre

ein

terv

entio

nson

qual

ityof

depr

essi

onca

re:

1)co

mpu

ter-

gene

rate

d,pa

per-

base

dre

-m

inde

rsfo

rde

pres

sion

with

trea

tmen

tre

com

men

datio

ns;

2)co

mpu

ter-

gene

rate

d,pa

-pe

r-ba

sed

rem

inde

rsw

ithdi

-ag

nosi

sin

form

atio

nal

one;

and

3)us

ualc

are

Effe

ctiv

enes

sSu

rvei

llanc

eA

cade

mic

Thre

eda

ysaf

ter

com

pute

rno

tific

a-tio

nof

poss

ible

moo

ddi

sord

er,

65%

ofph

ysic

ians

agre

edw

ithco

mpu

ter-

scre

ened

diag

nosi

s,13

%di

sagr

eed,

and

23%

wer

eun

cert

ain;

nodi

ffer

ence

sin

trea

t-m

ent

prov

ided

acro

ssgu

idel

ine-

expo

sure

cond

ition

Tim

eut

iliza

tion

/med

icat

ion

erro

rsK

oppe

let

al.,

2005

(91)

,m

ixed

quan

tita-

tive/

qual

itativ

ede

scrip

tive

met

hods

2002

–20

04C

POE

(Ecl

ipsy

sC

orp.

)In

patie

ntC

POE

infa

cilit

atin

gm

edic

atio

npr

escr

ibin

ger

rors

Safe

tyM

edic

atio

ner

rors

Aca

dem

icC

POE

faci

litat

ed22

type

sof

med

i-ca

tion

erro

rris

ks;

erro

rsw

ere

clas

sifie

das

bein

gdu

eto

1)fr

ag-

men

tatio

nof

data

and

failu

reto

inte

grat

eC

POE

syst

ems

with

othe

rho

spita

lsys

tem

san

d2)

flaw

sin

hum

an–m

achi

nein

ter-

face

Mek

hjia

net

al.

(88)

,20

02,

pre–

post

stud

y

2000

CPO

E/D

S(I

nvis

ion2

4,Si

emen

sC

orp.

)In

patie

nt/

ICU

CPO

Ew

ithel

ectr

onic

reco

rds

for

med

icat

ion

adm

inis

trat

ion

onca

rede

liver

ytim

e,w

orkf

low

proc

ess,

and

cost

s

Effic

ienc

yTi

me

utili

-za

tion/

med

ica-

tion

erro

rs

Aca

dem

ic64

%re

lativ

ede

crea

se(f

rom

328

to11

1m

in)

inm

edic

atio

ntu

rn-

arou

ndtim

e;43

%re

lativ

ede

-cr

ease

(fro

m45

7to

261

min

)in

com

plet

ion

time

for

radi

olog

ypr

oced

ures

;25

%re

lativ

ede

-cr

ease

(fro

m31

.3to

23.4

min

)in

repo

rtin

gtim

efo

rla

bora

tory

re-

sults

;11

.3–p

erce

ntag

epo

int

ab-

solu

tede

crea

se(f

rom

11.3

%to

0%)

intr

ansc

riptio

ner

rors

;5%

rela

tive

decr

ease

(fro

m3.

91to

3.71

d)in

seve

rity-

adju

sted

leng

thof

stay

;no

stat

istic

ally

sign

ifica

ntde

crea

ses

inov

eral

lco

st

W-14 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 www.annals.org

App

endi

xT

able

2—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

22)

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Type

ofIn

stit

utio

nK

eyFi

ndin

gs

Cor

dero

etal

.,20

04(8

9),

pre–

post

stud

yw

ithre

tros

pect

ive

revi

ew

2002

CPO

E/D

S(I

nvis

ion2

4,Si

emen

sC

orp.

)N

eona

tal

ICU

CPO

Ew

ithD

Son

med

icat

ion

erro

rsan

dca

rede

liver

ytim

ein

neon

atal

ICU

Safe

ty/e

ffic

ienc

yM

edic

atio

ner

rors

/tim

eut

ili-

zatio

n

Aca

dem

ic13

–per

cent

age

poin

tab

solu

tede

-cr

ease

(fro

m13

%to

0%)

inm

edic

atio

ndo

sing

erro

rs;

73%

rela

tive

decr

ease

(fro

m10

.5to

2.8

h)in

turn

arou

ndtim

efo

r1

med

icat

ion

(caf

fein

e);

24%

rela

-tiv

ede

crea

se(f

rom

42to

32m

in)

inra

diol

ogy

resp

onse

time;

phys

icia

nan

dst

aff

trai

ning

star

ted

4w

kbe

fore

CPO

Eim

ple-

men

tatio

n;nu

rse

lead

ers

rece

ived

16h

oftr

aini

ng,

nurs

esan

dcl

eri-

cals

taff

rece

ived

8h,

and

phys

i-ci

ans

rece

ived

2–4

h;du

ring

im-

plem

enta

tion,

info

rmat

ion

syst

ems

staf

fpr

ovid

ed24

-hsu

p-po

rtK

ilgor

eet

al.,

1998

(90)

,pr

e–po

stst

udy

1995

–199

6EH

R(C

areV

ue90

00,

Hew

lett

-Pac

kard

Cor

p.;

PIN

Syst

em,

Eclip

sys

Cor

p.)

ICU

Two

diff

eren

tnu

rsin

gIC

Uin

for-

mat

ion

syst

ems

with

CPO

Ean

dre

sults

repo

rtin

gon

nurs

ew

ork

patt

erns

and

cost

s

Effic

ienc

yTi

me

utili

-za

tion

Aca

dem

icEf

fect

ofco

mpu

ter

syst

ems

onnu

rse

char

ting

time

was

inco

n-cl

usiv

epe

rau

thor

sbe

caus

eIC

Upa

tient

cens

usva

ried;

staf

fsa

tis-

fact

ion

was

high

erw

ithC

areV

ue90

00th

anw

ithPI

NSy

stem

be-

caus

eof

inte

rfac

eea

sean

dgr

eate

rsu

ppor

tof

wor

kflo

w;

ex-

pect

edne

tan

nual

savi

ngs

from

pref

erre

dsy

stem

wer

ees

timat

edat

$320

359

Impl

emen

tati

onco

sts

Kra

ll,19

95(8

5),

desc

riptiv

equ

antit

ativ

est

udy

1994

EHR

(Epi

Car

e,Ep

icSy

stem

sC

orp.

)O

utpa

tient

EHR

use

onw

orkf

low

and

atti-

tude

sEf

ficie

ncy

Impl

emen

-ta

tion

cost

Kai

ser

Perm

anen

tePh

ysic

ians

took

30d

tore

turn

toba

selin

epr

oduc

tivity

leve

ls(p

a-tie

ntvi

sits

/d);

2-m

inin

crea

sein

phys

icia

ntim

epe

rvi

sit;

phys

icia

nsa

tisfa

ctio

nw

ithsy

stem

incr

ease

dov

ertim

e

Inte

rnal

lyde

vel-

oped

syst

ems

Adh

eren

ceK

hour

y,19

98(7

0),

time-

serie

sst

udy

1993

–199

7EH

R/D

SO

utpa

tient

EHR

with

DS

onad

here

nce

togu

idel

ine-

base

dca

reEf

fect

iven

ess/

effi-

cien

cyA

dher

ence

Kai

ser

Perm

anen

teA

dher

ence

togu

idel

ines

impr

oved

for

6co

nditi

ons;

leve

lsof

im-

prov

emen

tra

nged

from

4–to

52–p

erce

ntag

epo

int

abso

lute

incr

ease

sin

proc

ess

ofca

rede

liv-

ery;

estim

ated

annu

alsa

ving

s,$2

470

000

(cos

tof

syst

emde

velo

p-m

ent

not

incl

uded

)

Con

tinue

don

follo

win

gpa

ge

www.annals.org 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 W-15

App

endi

xT

able

2—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

22)

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Type

ofIn

stit

utio

nK

eyFi

ndin

gs

Orn

stei

net

al.,

1995

(72)

,pr

e–po

stst

udy

NS

EHR

/DS

Out

patie

ntEH

Rw

ithco

mpu

teriz

edre

min

d-er

son

deliv

ery

ofpr

even

tive

care

Effe

ctiv

enes

sA

dher

ence

Aca

dem

ic7

of7

coun

selin

gm

easu

res

im-

prov

ed:

abso

lute

incr

ease

inad

-he

renc

era

ngin

gfr

om13

to16

perc

enta

gepo

ints

;10

of15

scre

enin

gpr

oces

ses

impr

oved

:ab

solu

tein

crea

seap

prox

imat

ely

rang

ing

from

3to

20pe

rcen

tage

poin

tsG

arr

etal

.,19

93(7

1),

pre–

post

stud

y

1989

–199

0D

S/EH

RO

utpa

tient

EHR

with

com

pute

rized

rem

ind-

ers

onde

liver

yof

prev

entiv

eca

re

Effe

ctiv

enes

sA

dher

ence

Aca

dem

icA

bsol

ute

incr

ease

sin

deliv

ery

ofpr

even

tive

care

,1–

8pe

rcen

tage

poin

ts;

all5

serv

ices

incl

uded

Safr

anet

al.,

1995

(75)

,C

CT

1992

–199

3EH

R/D

SO

utpa

tient

Com

pute

rized

rem

inde

rsan

dal

erts

onde

liver

yof

HIV

care

Effe

ctiv

enes

sA

dher

ence

Aca

dem

icA

ppro

xim

ate

22–p

erce

ntag

epo

int

abso

lute

incr

ease

(fro

m46

%to

68%

)in

adhe

renc

eto

reco

m-

men

ded

proc

ess

ofca

reat

1y

afte

roc

curr

ence

ofcl

inic

alev

ent

war

rant

ing

rem

inde

r;29

–per

-ce

ntag

epo

int

abso

lute

incr

ease

(fro

m38

%to

67%

)in

phys

icia

nre

spon

ses

1m

oaf

ter

clin

ical

even

tw

arra

ntin

gal

ert

Schr

iger

etal

.,19

97(7

3),

CC

T19

92–1

995

DS/

EHR

Emer

genc

yde

part

-m

ent

Com

pute

rized

guid

elin

esem

-be

dded

inco

mpu

teriz

edch

artin

gsy

stem

desi

gned

totr

ack

5co

nditi

ons

onpr

o-ce

sses

ofca

refo

rex

posu

reof

heal

thca

rew

orke

rsto

bodi

lyflu

ids

Effe

ctiv

enes

s/ef

fi-ci

ency

Adh

eren

ceA

cade

mic

12–p

erce

ntag

epo

int

abso

lute

in-

crea

se(f

rom

83%

to96

%)

inad

here

nce

to5

trea

tmen

tgu

ide-

lines

;20

–per

cent

age

poin

tab

so-

lute

incr

ease

(fro

m63

%to

83%

)in

adhe

renc

eto

4gu

idel

ines

onla

bora

tory

test

use;

62–p

erce

nt-

age

poin

tab

solu

tein

crea

se(f

rom

31%

to93

%)

indo

cum

ente

dad

here

nce

toaf

terc

are

guid

e-lin

es;

42–p

erce

ntag

epo

int

abso

-lu

tein

crea

se(f

rom

57%

to98

%)

inad

here

nce

togu

idel

ines

for

docu

men

tatio

nof

patie

nthi

stor

y;al

lmea

sure

sde

crea

sed

tow

ard

base

line

rate

sw

hen

com

pute

rsy

stem

was

turn

edof

f;23

%re

l-at

ive

decr

ease

(fro

m$5

20to

$401

)in

tota

lper

-pat

ient

cost

s

Con

tinue

don

follo

win

gpa

ge

W-16 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 www.annals.org

App

endi

xT

able

2—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

22)

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Type

ofIn

stit

utio

nK

eyFi

ndin

gs

Schr

iger

etal

.,20

00(7

4),

CC

T

1992

–199

5D

S/EH

REm

erge

ncy

depa

rt-

men

t

Com

pute

rized

guid

elin

esem

-be

dded

inco

mpu

teriz

edch

artin

gsy

stem

desi

gned

totr

ack

5co

nditi

ons

onpr

o-ce

sses

ofca

refo

rev

alua

tion

offe

brile

child

ren

�ag

e3

ypr

esen

ting

toem

erge

ncy

de-

part

men

t

Effe

ctiv

enes

s/ef

fi-ci

ency

Adh

eren

ceA

cade

mic

Per

auth

ors’

repo

rt,

13–p

erce

ntag

epo

int

abso

lute

incr

ease

(fro

m80

%to

92%

)in

docu

men

ted

adhe

renc

eto

guid

elin

esfo

rm

ed-

ical

hist

ory

and

phys

ical

exam

ina-

tion;

33–p

erce

ntag

epo

int

abso

-lu

tein

crea

se(f

rom

48%

to81

%)

indo

cum

ente

dad

here

nce

togu

idel

ines

for

afte

rcar

ein

stru

c-tio

n;al

lrat

esre

turn

edto

base

line

whe

nco

mpu

ter

syst

emw

astu

rned

off;

nost

atis

tical

lysi

gnifi

-ca

ntdi

ffer

ence

sin

appr

opria

te-

ness

oftr

eatm

ent,

appr

opria

te-

ness

orut

iliza

tion

rate

sof

diag

nost

icte

sts,

orte

stch

arge

spe

rpa

tient

Day

etal

.,19

95(7

6),

pre–

post

stud

y

1992

–199

3D

S/EH

REm

erge

ncy

depa

rt-

men

t

Com

pute

rized

guid

elin

esem

-be

dded

inco

mpu

teriz

edch

artin

gsy

stem

desi

gned

totr

ack

5co

nditi

ons

onpr

o-ce

sses

ofca

refo

rlo

wer

back

pain

Effe

ctiv

enes

s/ef

fi-ci

ency

Adh

eren

ceA

cade

mic

No

stat

istic

ally

sign

ifica

ntdi

ffer

-en

ces

inap

prop

riate

ness

ofdi

ag-

nost

icte

stin

gor

trea

tmen

t;no

stat

istic

ally

sign

ifica

ntde

crea

sein

cost

s;12

–to

51–p

erce

ntag

epo

int

abso

lute

incr

ease

indo

cu-

men

tatio

nof

6m

edic

alhi

stor

yite

ms;

13–

to70

–per

cent

age

poin

tab

solu

tein

crea

sein

docu

-m

enta

tion

of6

afte

rcar

eco

unse

l-in

gite

ms

Sim

onet

al.,

2000

(77)

,R

CT

NS

EHR

/dat

asu

mm

a-ry

/DS

Out

patie

ntTh

ree

inte

rven

tions

:1)

feed

back

toph

ysic

ians

ofco

mpu

teriz

edda

tasu

mm

arie

sw

ithtr

eat-

men

tre

com

men

datio

nsfo

rde

pres

sion

care

,2)

com

pute

r-iz

edfe

edba

ckpl

uste

leph

one

follo

w-u

p,an

d3)

usua

lcar

e

Effe

ctiv

enes

s/ef

fi-ci

ency

Adh

eren

ceN

onac

adem

ic(G

roup

Hea

lthC

oope

rativ

eof

Puge

tSo

und)

No

stat

istic

ally

sign

ifica

ntdi

ffer

ence

inco

mpu

teriz

edD

Sgr

oup

com

-pa

red

with

usua

lcar

e;15

%re

la-

tive

impr

ovem

ent

(fro

m0.

83vs

.0.

98)

inde

pres

sion

scor

esin

com

pute

rized

DS

with

tele

phon

efo

llow

-up

grou

p,w

ithas

soci

ated

incr

ease

inlik

elih

ood

ofre

ceiv

ing

antid

epre

ssan

tth

erap

y;$8

0in

-cr

ease

inco

sts

inte

leph

one

fol-

low

-up

grou

p

Uti

lizat

ion

ofca

reBa

irdet

al.,

1984

(78)

,R

CT

NS

DS/

elec

tron

icpr

escr

ib-

ing

Out

patie

ntC

ompu

ter-

gene

rate

d,pa

per-

base

dph

arm

acy

rem

inde

rsfr

omho

spita

lmai

nfra

me–

base

din

form

atio

nsy

stem

onpr

escr

iptio

nre

fillr

ates

Acc

ess

Util

izat

ion

ofca

reA

cade

mic

No

stat

istic

ally

sign

ifica

ntdi

ffer

ence

inre

fillr

ates

;in

itial

deve

lopm

ent

ofso

ftw

are

prog

ram

,$2

00;

cost

per

day

toge

nera

tere

min

ders

,$1

4

Con

tinue

don

follo

win

gpa

ge

www.annals.org 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 W-17

App

endi

xT

able

2—C

onti

nued

Stud

y,Y

ear

(Ref

eren

ce),

Type

ofSt

udy

(n�

22)

Dat

aC

olle

ctio

nPr

imar

yH

ITIn

terv

enti

onSe

ttin

gPu

rpos

e(T

oD

eter

min

eth

eEf

fect

of..

.)D

imen

sion

sof

Car

eEn

dPo

ints

Effe

ctEv

alua

ted

Type

ofIn

stit

utio

nK

eyFi

ndin

gs

Sand

ers

and

Mill

er,

2001

(79)

,pr

e–po

stst

udy

2000

–200

1D

S/C

POE

Inpa

tient

Com

pute

rized

guid

elin

esin

te-

grat

edin

toa

CPO

Esy

stem

onut

iliza

tion

ofC

Tan

dM

RI

Effic

ienc

yU

tiliz

atio

nof

care

Aca

dem

ic5%

rela

tive

decr

ease

inne

uror

adi-

olog

yC

Tan

dM

RI

diag

nost

icte

stin

g;40

%of

user

sre

ceiv

ing

com

pute

rized

guid

elin

eor

dere

da

nonr

ecom

men

ded

test

Surv

eilla

nce

Wel

lset

al.,

2003

(81)

,pr

e–po

stst

udy

2001

–200

2D

S/EH

RO

utpa

tient

EHR

onid

entif

ying

patie

nts

tak-

ing

2dr

ugs

for

whi

cha

com

-bi

natio

npi

llis

avai

labl

ean

dth

enge

nera

ting

com

pute

rized

rem

inde

rsto

prom

ote

com

bi-

natio

nth

erap

y

Effe

ctiv

enes

sSu

rvei

l-la

nce/

adhe

r-en

ce

Aca

dem

ic27

%of

patie

nts

elig

ible

wer

esw

itche

dto

com

bina

tion

ther

apy;

cost

savi

ngs

rela

ted

toco

mbi

ned

ther

apy

tota

led

$615

9pe

rye

ar;

241

patie

nts

wer

esc

reen

edas

elig

ible

for

com

bina

tion

ther

apy;

tota

lnum

ber

ofun

ique

patie

nts

seen

durin

gst

udy

not

give

n

Med

icat

ion

erro

rsPo

tts

etal

.,20

04(8

0),

pre–

post

stud

y

2001

–200

2C

POE/

DS

ICU

CPO

Ew

ithD

Son

med

icat

ion

erro

rsin

pedi

atric

ICU

Effic

ienc

yM

edic

atio

ner

rors

Aca

dem

ic41

%re

lativ

ede

crea

se(f

rom

2.2

erro

rs/1

00or

ders

to1.

3er

rors

/10

0or

ders

)in

med

icat

ion

erro

rsca

tego

rized

aspo

tent

iala

dver

sedr

ugev

ents

;96

%re

lativ

ede

-cr

ease

(fro

m30

erro

rs/1

00or

-de

rsto

0.2

erro

r/10

0or

ders

)in

med

icat

ion

pres

crib

ing

orde

rs;

decr

ease

soc

curr

edin

allc

ateg

o-rie

sof

med

icat

ion

erro

rs

Impl

emen

tati

onco

stK

hour

y,19

97(8

2),

time-

serie

sst

udy

1989

–NS

EHR

Out

patie

ntLo

ng-t

erm

cost

san

dbe

nefit

sof

impl

emen

ted

EHR

Effic

ienc

yIm

plem

en-

tatio

nco

st

Kai

ser

Perm

anen

teC

ost

ofde

velo

pmen

tes

timat

edat

$10

mill

ion;

proj

ect

took

8y

from

begi

nnin

gof

deve

lopm

ent

tofu

llim

plem

enta

tion;

tota

lon-

goin

gan

nual

expe

nses

estim

ated

tobe

$1.1

mill

ion

per

year

;ex

-pe

cted

savi

ngs

per

year

esti-

mat

edas

$3.7

mill

ion,

with

grea

test

savi

ngs

from

redu

ctio

nin

med

ical

reco

rdro

omst

aff;

sys-

tem

pred

icte

dto

pay

for

itsel

fin

year

13

*C

ity

and

stat

elo

cati

ons

ofm

anuf

actu

rers

are

asfo

llow

s:M

edic

aLog

icC

orp.

,B

eave

rton

,O

rego

n;E

pic

Syst

ems

Cor

p.,

Ver

ona,

Wis

cons

in;

Ecl

ipsy

sC

orp.

,B

oca

Rat

on,

Flor

ida;

Siem

ens

Cor

p.,

New

Yor

k,N

ewY

ork;

Hew

lett

-Pac

kard

Cor

p.,P

alo

Alto

,Cal

iforn

ia.C

CT

�co

ntro

lled

clin

ical

tria

l;C

POE

�co

mpu

teri

zed

prov

ider

orde

ren

try;

CT

�co

mpu

ted

tom

ogra

phy;

DS

�de

cisi

onsu

ppor

t;E

HR

�el

ectr

onic

heal

thre

cord

;HIT

�he

alth

info

rmat

ion

tech

nolo

gy;

ICU

�in

tens

ive

care

unit

;M

RI

�m

agne

tic

reso

nanc

eim

agin

g;N

S�

not

spec

ified

;R

CT

�ra

ndom

ized

,co

ntro

lled

tria

l.

W-18 16 May 2006 Annals of Internal Medicine Volume 144 • Number 10 www.annals.org