+ All Categories
Home > Documents > More Medicine, Fewer Clicks: How Informatics Can Actually ... · mendous resources in developing...

More Medicine, Fewer Clicks: How Informatics Can Actually ... · mendous resources in developing...

Date post: 29-May-2020
Category:
Upload: others
View: 1 times
Download: 0 times
Share this document with a friend
10
450 2017 ASCO EDUCATIONAL BOOK | asco.org/edbook A s cancer specialists, we have grown up in an informaon age in which we expect data systems to make us more effecve and more efficient. There has been an unprece- dented improvement in cancer death rates over the past 25 years. 1 We expect machine learning and other informacs innovaons to help us advance the quality of care even faster, and not simply result in addional boxes to check in an EHR. In 2017, there are growing concerns that these expecta- tions have not been met, to the point at which some clini- cians are cing dissasfacon with health informaon tech- nology as a major driver of job dissasfacon. 2 Despite this negavity, many tools are in development or operaonal for use in the clinic today to help make us beer at what we do. Recently, there has been much interest in facilitang data systems to become more integrated and interoperable and to deliver care faster. These are recurring themes within the ASCO's parcipaon in the Cancer Moonshot work and in the 21st Century Cures legislaon that passed in 2016. 3 The 2016 President’s Cancer Panel report, Improving Cancer-Re- lated Outcomes With Connected Health, states, “We live at a most excing and crical me of technological advances with potenal to help individuals manage and improve their own health and support high-quality, paent-centered can- cer care.” 4 In this arcle, we explore some of the success that in- formacs can bring to the pracce of oncology. First, we review some of the currently exisng informacs capabili- es at one author’s large integrated pracce, US Oncology. Second, we discuss the topic of incorporang external knowl- edge into oncology pracce and how informacs can pro- vide point-of-care soluons. Third, we discuss the challenges of clinical documentaon in the 21st century and how infor- macs tools can be used to make sense of messy real-world data. Finally, we turn toward patients and discuss how new technologies are emerging to enable digital donaons to research. CREATING AN INFORMATICS-ENABLED ONCOLOGY PRACTICE Big Data Over the past decade, we have seen advancements in big data systems that allow us to aggregate data in cancer be- yond what has tradionally been collected by the cancer registry system. This is used across systems of care deliv- ery to understand outcomes in various disease states out- side of clinical trials in the form of real-world evidence. It remains a limitaon in adult oncology that only 2% to 3% of paents enroll in prospecve clinical trials, and age- and PATT ET AL More Medicine, Fewer Clicks: How Informacs Can Actually Help Your Pracce Debra A. Pa, MD, MPH, MBA, Elmer V. Bernstam, MD, MSE, MS, FACP, FACMI, Joshua C. Mandel, MD, David A. Kreda, and Jeremy L. Warner, MD, MS OVERVIEW In the informaon age, we expect data systems to make us more effecve and efficient—not to make our lives more dif- ficult! In this arcle, we discuss how we are using data systems, such as electronic health records (EHRs), to improve care delivery. We illustrate how US Oncology is beginning to use real-world evidence to facilitate trial accrual by automac idenficaon of eligible paents and how big data and predicve analycs will transform the field of oncology. Some infor- maon systems are already being used at the point of care and are already empowering clinicians to improve the care of their paents in real me. Telehealth plaorms are being used to bridge gaps that currently exist in experse, geography, and technical capability. Opmizing virtual collaboraon, such as through virtual tumor boards, is empowering commu- nies that are geographically disparate to coordinate care. Informacs methods can provide soluons to the challenging problems of how to manage the vast amounts of data confronng the praccing oncologist, including informaon about treatment regimens, side effects, and the influence of genomics on the pracce of oncology. We also discuss some of the challenges of clinical documentaon in the modern era, and review emerging efforts to engage paents as digital donors of their EHR data. From Texas Oncology, Ausn, TX, McKesson Specialty Health and the US Oncology Network, The Woodlands, TX; School of Biomedical Informacs, The University of Texas Health Science Center, Houston, TX; Verily (Google Life Sciences), USA Research Faculty, Harvard Medical School, Cambridge, MA; Vanderbilt University, Vanderbilt Cancer Registry, Nashville, TN. Disclosures of potenal conflicts of interest provided by the authors are available with the online arcle at asco.org/edbook. Corresponding author: Jeremy L. Warner, MD, MS, Vanderbilt University, 2220 Pierce Ave., 777 PRB, Nashville, TN 37232; email: [email protected]. © 2017 American Society of Clinical Oncology
Transcript
Page 1: More Medicine, Fewer Clicks: How Informatics Can Actually ... · mendous resources in developing solutions for better use of cancer data include TriNetX, McKesson Specialty Health,

450 2017 ASCO EDUCATIONAL BOOK | asco.org/edbook

As cancer specialists, we have grown up in an information age in which we expect data systems to make us more

effective and more efficient. There has been an unprece-dented improvement in cancer death rates over the past 25 years.1 We expect machine learning and other informatics innovations to help us advance the quality of care even faster, and not simply result in additional boxes to check in an EHR. In 2017, there are growing concerns that these expecta-tions have not been met, to the point at which some clini-cians are citing dissatisfaction with health information tech-nology as a major driver of job dissatisfaction.2 Despite this negativity, many tools are in development or operational for use in the clinic today to help make us better at what we do. Recently, there has been much interest in facilitating data systems to become more integrated and interoperable and to deliver care faster. These are recurring themes within the ASCO's participation in the Cancer Moonshot work and in the 21st Century Cures legislation that passed in 2016.3 The 2016 President’s Cancer Panel report, Improving Cancer-Re-lated Outcomes With Connected Health, states, “We live at a most exciting and critical time of technological advances with potential to help individuals manage and improve their own health and support high-quality, patient-centered can-cer care.”4

In this article, we explore some of the success that in-formatics can bring to the practice of oncology. First, we review some of the currently existing informatics capabili-ties at one author’s large integrated practice, US Oncology. Second, we discuss the topic of incorporating external knowl-edge into oncology practice and how informatics can pro-vide point-of-care solutions. Third, we discuss the challenges of clinical documentation in the 21st century and how infor-matics tools can be used to make sense of messy real-world data. Finally, we turn toward patients and discuss how new technologies are emerging to enable digital donations to research.

CREATING AN INFORMATICS-ENABLED ONCOLOGY PRACTICEBig DataOver the past decade, we have seen advancements in big data systems that allow us to aggregate data in cancer be-yond what has traditionally been collected by the cancer registry system. This is used across systems of care deliv-ery to understand outcomes in various disease states out-side of clinical trials in the form of real-world evidence. It remains a limitation in adult oncology that only 2% to 3% of patients enroll in prospective clinical trials, and age- and

PATT ET AL

More Medicine, Fewer Clicks: How Informatics Can Actually Help Your PracticeDebra A. Patt, MD, MPH, MBA, Elmer V. Bernstam, MD, MSE, MS, FACP, FACMI, Joshua C. Mandel, MD, David A. Kreda, and Jeremy L. Warner, MD, MS

OVERVIEW

In the information age, we expect data systems to make us more effective and efficient—not to make our lives more dif-ficult! In this article, we discuss how we are using data systems, such as electronic health records (EHRs), to improve care delivery. We illustrate how US Oncology is beginning to use real-world evidence to facilitate trial accrual by automatic identification of eligible patients and how big data and predictive analytics will transform the field of oncology. Some infor-mation systems are already being used at the point of care and are already empowering clinicians to improve the care of their patients in real time. Telehealth platforms are being used to bridge gaps that currently exist in expertise, geography, and technical capability. Optimizing virtual collaboration, such as through virtual tumor boards, is empowering commu-nities that are geographically disparate to coordinate care. Informatics methods can provide solutions to the challenging problems of how to manage the vast amounts of data confronting the practicing oncologist, including information about treatment regimens, side effects, and the influence of genomics on the practice of oncology. We also discuss some of the challenges of clinical documentation in the modern era, and review emerging efforts to engage patients as digital donors of their EHR data.

From Texas Oncology, Austin, TX, McKesson Specialty Health and the US Oncology Network, The Woodlands, TX; School of Biomedical Informatics, The University of Texas Health Science Center, Houston, TX; Verily (Google Life Sciences), USA Research Faculty, Harvard Medical School, Cambridge, MA; Vanderbilt University, Vanderbilt Cancer Registry, Nashville, TN.

Disclosures of potential conflicts of interest provided by the authors are available with the online article at asco.org/edbook.

Corresponding author: Jeremy L. Warner, MD, MS, Vanderbilt University, 2220 Pierce Ave., 777 PRB, Nashville, TN 37232; email: [email protected].

© 2017 American Society of Clinical Oncology

Page 2: More Medicine, Fewer Clicks: How Informatics Can Actually ... · mendous resources in developing solutions for better use of cancer data include TriNetX, McKesson Specialty Health,

asco.org/edbook | 2017 ASCO EDUCATIONAL BOOK  451

HOW INFORMATICS CAN HELP YOUR PRACTICE

race-related disparities persist, yet we would like to learn from all patients with cancer.5,6 There are also some diseases and novel molecular mechanisms in cancer that make clin-ical trials a challenge to open and accrue patients appro-priately because of their rarity. Big data systems are a good answer to these challenges as they allow us to look for the needles within the haystacks and learn collectively. Some real-world evidence is also being used in other countries and is now even being considered by the U.S. Food and Drug Administration (FDA) for drug approval, with some caveats.7 The FDA issued draft guidance about the use of real-world evidence to support regulatory decisions around medical devices in 2016.8 Using real-world data systems to screen patients within care delivery systems, patients with rare diseases can be identified for clinical trials with just-in-time mechanisms including opening clinical trials at the site of care when these patients are identified. Several big data aggregators are pioneers in this new landscape: (1) Cancer-LinQ, ASCO’s data aggregation and sharing platform among many EHR sources to enlighten outcomes and report quality in cancer care9,10; (2) the National Cancer Institute’s Genomic Data Commons (https://gdc.cancer.gov); (3) the Oncology Research Information Exchange Network, a collaboration of many prominent North American cancer centers; (4) the American Association for Cancer Research’s Project Genom-ics Evidence Neoplasia Information Exchange; (5) the Trian-gle Census Research Network at Duke University, informing data aggregation and dissemination; and (6) Project Data Sphere. Additional private entities that have invested tre-mendous resources in developing solutions for better use of cancer data include TriNetX, McKesson Specialty Health,

Flatiron Health, and IBM Watson Health, among many oth-ers. Other systems that are internal to organizations are integrating molecular data to identify patients for selec-tion for clinical trials. For example, Syapse is a commercial partner that works within health systems to implement precision medicine programs. US Oncology research has a just-in-time mechanism of clinical trial initiation called the STAR program to accrue patients when identified with n-of-1 tumors that would otherwise be hard to accrue in inde-pendent systems.

Clinical decision support systems (CDSS) have become integrated in cancer care in many ways: facilitating com-pliance among clinicians in prescribing therapeutic inter-ventions within guidelines of care delivery (a quality en-hancement), facilitating screening for research accrual for appropriately selected patients, and using apps and other forms of digital patient engagement to inform patients to act on, alter, or contact their providers regarding their plans of care given their individualized data. Across a large net-work of US Oncology community practices that vary from urban and suburban to rural and frontier in their locations, a CDSS platform called Clear Value Plus was implemented, providing an interface at the point of service for chemother-apy ordering in a value-based mechanism within nationally accepted guidelines. The implementation of this CDSS sig-nificantly improved reportable data, guideline compliance, and exception reporting, making therapy decisions easier for doctors at the point of service, in addition to enhancing guideline compliance for patients and providing the neces-sary data to practices to proceed with prior authorization with payers, thus enhancing quality and time efficiencies.11 We fully acknowledge the real concerns regarding alert fa-tigue in the implementation of CDSS, and strongly encour-age the efforts being made in the field of human-computer interaction to improve this experience.

Predictive analytics platforms are being used to improve outcomes in patients with cancer. Data from EHRs and other data sources have been used to develop models to predict the risk for hospitalization in other diseases. For example, in populations with low-socioeconomic status and heart fail-ure, risk prediction and the interventions based upon it re-duced hospitalization risk in a high-risk and difficult-to-treat population.12 Warner et al have previously described an EHR-based predictive model for hospital-acquired compli-cations.13 Using such models to inform the care team about risk and facilitate appropriate interventions may be effec-tive at reducing hospitalizations and readmissions in high-risk groups. Similar models have been developed for high-risk cancers and are being used to inform clinicians about risk and facilitate support systems and care interventions to reduce the risk for hospitalization accordingly.14 Prediction of treatment intolerance, which can lead to nonadherence, especially in novel therapies, is also an evolving area of re-search, although the prior body of evidence suggests that side effects are a common reason for nonadherence and early discontinuation for traditional therapies as well.15 Given that recent studies suggest that treatment discontinuation

KEY POINTS

• As cancer specialists, we have grown up in an information age in which we expect data systems to make us more effective and more efficient; despite recent concerns surrounding health information technology, we are convinced that there is significant potential that is yet to be met on the large scale.

• US Oncology is a large integrated practice that has implemented big data, predictive analytics, and telehealth applications at the point of care.

• Factual and contextual knowledge, especially regarding the interpretation of genomic sequence variation in cancer, will require external knowledge support that can be integrated into clinician workflow using emerging informatics technologies.

• The challenges of clinical documentation in the 21st century are significant because of increasing care complexity and regulatory and billing requirements, but informatics technologies exist to facilitate documentation and its secondary use.

• An emerging technology called Sync for Science will be piloted in 2017, with a goal of enabling patients to become digital donors for EHR-based research such as that envisioned by the Precision Medicine Initiative’s All of Us research program.

Page 3: More Medicine, Fewer Clicks: How Informatics Can Actually ... · mendous resources in developing solutions for better use of cancer data include TriNetX, McKesson Specialty Health,

452 2017 ASCO EDUCATIONAL BOOK | asco.org/edbook

PATT ET AL

because of intolerance is the most common reason for rev-olutionary drugs such as ibrutinib,16,17 it will be critically im-portant to identify vulnerable populations. Presently, pre-dictive analytic platforms can come across the EHR in the form of CDSS so that they are available to clinicians at the point of care (Fig. 1).

Telehealth and virtual collaboration platforms are another way US Oncology uses data systems to enhance care de-livery with efficiency. Use of these platforms is growing in scope, scale, and prevalence throughout the United States, and many states are currently considering policies that in-fluence their implementation.18 Innovations in platforms of interaction telehealth and virtual collaboration allow us to bridge existing gaps in geography and expertise. In the US Oncology network of oncology practices, there are sites of service that vary from urban and suburban to rural and fron-tier, and staffing and subspecialty expertise is also variable.

Telehealth platforms have allowed for consultation with subspecialty experts in neuro-oncology and genetic risk as-sessment that otherwise would have required a drive of sev-eral hours, a geographic barrier to care that frequently re-sults in diminished utilization of subspecialty services. This has allowed patients to access subspecialty services and treatments they otherwise would not have access to and it helps us make quality care global. Our present abilities to implement telehealth include multimodal virtual collabora-tion (between clinicians or between clinicians and patients) and remote review of imaging and pathology, but also have become enhanced in our ability to complement the physi-cal examination with universal serial bus attachments such as sphygmomanometers, stethoscopes, ophthalmoscopes, electrocardiographs, ultrasound probes, and cameras to make the skin examination more sensitive than to the hu-man eye. Teledermatoscopy programs have been imple-mented to diagnose and follow at-risk skin lesions to detect

early melanoma.19 There are even multimodal fiber optic probes that can be used at remote sites with a clinician’s assistance to interrogate cutaneous lesions and replace the need for some biopsies in skin cancer.20 How we interact with these systems continues to change over time. These are not only technological advancements but also ways we must think differently about supporting clinical workflow to optimize the patient experience as our technological ca-pabilities grow. Optimizing virtual collaboration with all of these platforms also allows virtual multidisciplinary plan-ning, which can often be a critical quality measure in plan-ning cancer care treatment. Virtual tumor boards are now in existence in many networks today, allowing geographically disparate multidisciplinary planning.21 All of these systems are tools that close physical and mental gaps that limit care delivery today.

INTEGRATING EXTERNAL KNOWLEDGE INTO THE ONCOLOGY WORKFLOWGenerally speaking, there are four types of knowledge per-tinent to the day-to-day practice of oncology: procedural, transactional, factual, and contextual. The first category, procedural knowledge, includes those factors pertinent to daily practice and is usually specific to a given location. Ex-amples may include (1) what antibiotics and antiemetics are available in the hospital formulary, (2) the times and days laboratory technicians are available to assist with bone mar-row biopsy and aspiration procedures, and (3) standard pro-tocols decided by consensus or disease group leadership. Often, procedural knowledge is kept locally in the form of standard operating procedures.

The next category, transactional knowledge, includes those factors pertinent to the business aspects of oncology practice. This includes knowledge about what billing codes (e.g., International Classification of Diseases, 10th Revision,

FIGURE 1. Predictive Analytics to Understand Risk

Multiple sources of data are used to create a risk model that can predict likely outcomes for individual patients, such as the risk for hospitalization after the administration of chemotherapy.

Page 4: More Medicine, Fewer Clicks: How Informatics Can Actually ... · mendous resources in developing solutions for better use of cancer data include TriNetX, McKesson Specialty Health,

asco.org/edbook | 2017 ASCO EDUCATIONAL BOOK  453

HOW INFORMATICS CAN HELP YOUR PRACTICE

Clinical Modification) are necessary and sufficient to justify a given level of professional billing, what billing codes will translate into an appropriate diagnosis-related group for a given hospitalization, and details of negotiated contracts with third-party insurers and pharmaceutical companies. As with procedural knowledge, most transactional knowledge is location specific, although some, such as information about International Classification of Diseases codes, may be amenable to external knowledge resources.

The last two categories are inter-related. Factual knowl-edge includes information about a disease, a prognosis, and associated treatments. One is likely to find this type of knowledge in an encyclopedia or a medical textbook. Im-portantly, factual knowledge is by convention limited to a representative example or a range of commonly expected examples. In other words, factual knowledge is generic and often not applicable to an individual patient. On the other hand, contextual knowledge takes into account features of an individual patient and is necessary (although not suffi-cient) for the practice of precision or personalized medi-cine.22 In oncology practice, context includes comorbidities; performance status; treatment history, including prior drug exposures, length and depth of response, and pertinent adverse events; behavioral determinants of health such as substance abuse; psychological distress and psychosocial support systems; and belief systems (e.g., Jehovah’s Wit-nesses will not accept blood transfusions, which may influ-ence decisions about the intensity of cytotoxic chemother-apy).

The crux of the issue of knowledge management in on-cology is what proportion of knowledge is internal as op-posed to external. Internalized knowledge is that which is available to a practitioner through memory, with or without prompting. Externalized knowledge is that which is available through any type of ancillary resource. Internal knowledge is not to be taken lightly; after all, much of the 4 years of medical school and 5 to 7 years of postgraduate training are focused on the tasks of acquiring and retaining knowledge. Nevertheless, it was observed many decades ago that a practicing clinician could not possibly grasp the totality of medical knowledge.23 We have previously determined with some back-of-the-envelope calculations that it would be necessary to read 272 articles each day, 365 days a year, just to keep up with the cancer literature.24 So how is a practic-ing clinician to approach and master both facts and context?

Historically, the incorporation of external knowledge into the clinical workflow falls under CDSS. CDSS is part of the lingua franca of biomedical informatics and features prom-inently in recent regulations, including the meaningful-use rules and the FDA’s guidance on the regulation of mobile medical applications.25,26 Interestingly, one of the earliest CDSS, ONCOCIN,27 was an oncology protocol management system. Indeed, the selection of chemotherapy proto-cols and tracking of their dosing parameters is one of the areas most amenable to external knowledge support. Al-though certain EHRs provide the means to build and main-tain local chemotherapy regimen libraries, these are rarely

comprehensive. In 2011, Dr. Peter Yang founded the site HemOnc.org, with the goal of creating a freely available, comprehensive, and accurate resource for chemotherapy regimen details.28 The site listed more than 1,000 regimens by mid-2013, and, as of February 2017, HemOnc.org listed more than 2,000 disease-specific chemotherapy regimens across 84 distinct solid oncology, benign, and malignant hematology conditions; to our knowledge, it is the largest resource of its kind. Over time, the initial focus on capturing details of dosing and timing of chemotherapeutics has ex-panded to also include information on comparative efficacy and toxicity for randomized controlled trials and overall re-sponse rates for nonrandomized studies (Fig. 2).

HemOnc.org and similar resources can offer the practicing oncologist the ready means to bring external knowledge to bear, especially when prescribing obscure or infrequently used regimens. Another solution to this knowledge management problem is pathways. Pathways take into account cost, reim-bursement, efficacy, and the likelihood of treatment-related complications to varying degrees. Some, such as the National Comprehensive Cancer Network guidelines, provide a fair amount of latitude in treatment selection; others, including several vendor products, enforce treatment choices with po-tential penalties for overrides. Zon et al29 recently criticized the pathway approach for lacking clear processes, placing additional administrative burdens on oncology practices and not yet clearly demonstrating an impact on patient health outcomes.

The other area most in need of external knowledge sup-port is genomically guided treatment. This is a knowledge space that is simply too large to manage without assistance. Knowledge support in this evolving area is taking several forms: (1) extensively curated somatic panel test reports, (2) molecular tumor boards that convene experts either locally or remotely, and (3) genomic knowledge bases. Although cu-rated reports are critical, they suffer from two major flaws: (1) they are a snapshot from the time when the test was obtained and will not reflect the new genotype-phenotype knowledge that is constantly emerging, and (2) they are subject to considerable variation, as recently demonstrated by Balmaña et al.30 Molecular tumor boards can be both clinically useful and educational but do not necessarily fall within normal clinic workflows. Genomic knowledge bases hold great promise but currently lack uniformity in format and interpretation. Recently, Ritter et al,31 on behalf of the ClinGen Somatic Cancer Working Group, described a con-sensus for minimum variant level data, which is followed by knowledge bases such as My Cancer Genome32 and ClinGen.33 The FDA has recently issued draft guidance on the use of public human genetic variant databases to support clinical validity of next-generation sequencing panels.34 ASCO, the Association for Molecular Pathology, and the College of American Pathologists recently issued a unified set of stan-dards and guidelines for the interpretation and reporting of sequence variants in cancer. These efforts should eventually improve the uniformity of genomic test results.35

Once genomic data are integrated into the EHR, the ca-pacity for further innovation expands.36 In particular, a new

Page 5: More Medicine, Fewer Clicks: How Informatics Can Actually ... · mendous resources in developing solutions for better use of cancer data include TriNetX, McKesson Specialty Health,

454 2017 ASCO EDUCATIONAL BOOK | asco.org/edbook

PATT ET AL

standard produced by Health Level Seven International, called the Fast Healthcare Interoperability Resources (FHIR) standard, increases the ability to bring external knowledge, including genomics, into the clinical workflow. Warner et al previously demonstrated a FHIR-based app, called SMART Precision Cancer Medicine, that links users to factual ge-nomic knowledge (via Gene Wiki), contextual genomic knowledge (via My Cancer Genome), and contextual che-motherapy regimen knowledge (via HemOnc.org).37 This app was designed to launch from a tablet device with the intent of seamless integration into the workflow of a busy clinic; it and similar apps can also easily be integrated into certain EHR environments to provide a seamless user ex-perience.38 ASCO is currently investigating the possibility of creating an app that will bring the results from multiple ge-nomic knowledge bases to clinician users.

It is clear that clinical decision support, especially the invasive variety that disrupts workflow through alerts and reminders, can be perceived negatively. In anticipation of a backlash, Bates et al39 produced the seminal article “Ten Commandments for Effective Clinical Decision Support: Making the Practice of Evidence-Based Medicine a Reality” in 2003. This group and others have also documented the frequent practice of overriding alerts, even when the result may be a fatal drug interaction.40,41 Nevertheless, it is likely that clinical decision support and passive knowledge sup-port will increasingly become available within the clinical workflow, ideally in the form of apps clinicians can select and customize to meet their needs.42

The final issue that must be addressed is the accuracy of knowledge. Although there is no shortage of studies demon-strating the fallibility and malleability of internal knowledge

(the seminal paper by Tversky and Kahneman43 is an excel-lent primer), the failure of accuracy of external knowledge is often treated more harshly. This is likely an issue of trust more than anything. Failure of internal knowledge banks may be attributable to a variety of factors, but it is the rare practitioner who has a fundamental lack of trust in his or her own knowledge. Conversely, external knowledge that is incorrect and provably false can raise issues of trust pertain-ing to the entire knowledge base. This phenomenon is well demonstrated by the ongoing skepticism of the Wikipedia resource, despite academic publications showing high lev-els of accuracy in certain areas of the medical domain.44-47 Various approaches have been used to increase trust in ex-ternal knowledge bases, especially those that are openly collaborative: (1) clear attribution of content to well-known experts, (2) restriction of content creation to vetted indi-viduals, and (3) stamps of approval from specialty societies or other agencies. It remains to be seen which of these ap-proaches, or a combination thereof, will be most successful in gaining the trust of the user community.

THE CHALLENGES OF CLINICAL DOCUMENTATION IN THE 21ST CENTURYClinical documentation has always served multiple purposes, including clinical (record of clinical reasoning, decisions, and clinically relevant events), billing and financial (justify-ing payment for services rendered), and legal (What hap-pened? Who knew what and when?). Over time, practices and processes evolved that variably addressed all of these purposes. Some of these processes and practices were formal, but some were informal and specific to individual clinicians. With the implementation of EHRs, many of

FIGURE 2. A Portion of the Docetaxel for Non–Small Cell Lung Cancer Regimen on HemOnc.org, Showing Comparative Efficacy for 11 Randomized Controlled Trials

Hyperlinks under "Study" link to the original articles; those under "Comparator" link to other regimens on HemOnc.org.Abbreviations: IV, intravenous; OS, overall survival; PFS, progression-free survival.

Page 6: More Medicine, Fewer Clicks: How Informatics Can Actually ... · mendous resources in developing solutions for better use of cancer data include TriNetX, McKesson Specialty Health,

asco.org/edbook | 2017 ASCO EDUCATIONAL BOOK  455

HOW INFORMATICS CAN HELP YOUR PRACTICE

these processes were changed, either consciously or un-consciously. Although there are clear benefits of comput-erizing health care, there are also a number of challenges, particularly related to documentation. In this section, we discuss three challenges posed by the computerization of clinical documentation and the changing health care envi-ronment.

Structuring Inherently Messy Clinical RealityThe first challenge relates to a fundamental mismatch be-tween our tools and what we want to accomplish. Clin-ical reality is difficult to define precisely. Consider this simple example: Is hypertension a disease of the blood vessels? Heart? Kidneys? Brain? Clearly all are affected and involved. However, putting hypertension into a cat-egory with well-defined necessary and sufficient condi-tions is challenging. Further, clinical concepts are con-tinuously evolving. For example, the definition of a gene has changed dramatically as our understanding of mo-lecular biology has improved. Similarly, genomically in-formed therapy is constantly evolving. Today’s variant of unknown significance may be an actionable (targetable) variant tomorrow.

With paper records, we were not tied to specific cat-egories. We recorded our thoughts using natural language rather than trying to express our thoughts using a pre-defined set of categories that are often inadequate to repre-sent our intended meaning. Freehand drawings or diagrams could be inserted where appropriate, and shorthand was widely used. Rosenbloom et al48 published an overview of the tensions between structured and unstructured clinical documentation.

There is no easy answer to this challenge. However, there are promising developments. First, natural language pro-cessing technology can be very useful when 100% accuracy is not required. For example, algorithms can identify specific concepts in the text even when they are not referenced with a particular name (e.g., breast cancer, brst ca, and IDC [inva-sive ductal carcinoma] can be recognized as synonymous). This can be very useful for a variety of purposes, including identifying cases of a particular malignancy, cancer stage, or treatment outcome in large clinical data sets and for auto-matically summarizing complex patient histories. A review of natural language processing in oncology was recently published in JAMA Oncology.49

Second, the field of human-computer interaction has de-veloped into an engineering discipline with validated ap-proaches to matching users and tasks (e.g., a clinician who has to write a note documenting an office visit) to specific tools and their characteristics. Usability experts can define existing workflow, identify areas that can be improved, and guide implementation of systems that match user needs. Professional organizations have recognized that improving the usability of clinical systems has the potential to improve clinical outcomes (e.g., by reducing errors) and have pub-lished recommendations for incorporating usability into the design of clinical systems.50

Competing Priorities (Business Versus Clinical Needs)In many important ways, health care is a business. Institu-tions are reasonably concerned about their financial perfor-mance and must comply with an increasing regulatory bur-den. As a result, the decision to implement a clinical system is often driven more by business concerns rather than clin-ical needs, for example, the need to document compliance and streamline financial (billing) operations.

Clinical reimbursement has traditionally relied on docu-mentation of specific services rendered. Thus, clinical notes now contain specific billing-oriented phrases such as “40 minutes spent at the bedside with greater than 50% of this time spent on counseling.” This, along with the need to doc-ument in increasing detail to justify specific service levels, has led to administratively compliant but clinically less use-ful documentation.

Further, computerized physician order entry is a very ef-fective way to track and influence clinical behavior. An un-desirable behavior (e.g., daily laboratories) can simply be made inconvenient to order (e.g., by requiring a daily writ-ten justification). Thus, health information technology has increased the ability of the business enterprise to monitor and influence the clinical enterprise without assuming di-rect responsibility for clinical outcomes.

This challenge is primarily social, rather than technical. For a variety of reasons, clinicians have been reluctant ad-vocates for clinical priorities. As a result, business priorities may outweigh clinical priorities at times simply because the clinical enterprise lacks effective representation when the relevant decisions are being made. To their credit, organi-zations increasingly recognize the need for clinical champi-ons in board rooms and are hiring clinician-informaticians to lead clinical information technology efforts (e.g., chief medical information officers).

Ease of Creating Data Versus Useful InformationCurrent computer technology makes it very easy to gener-ate and replicate data. For example, with a few clicks, one can copy and paste radiology reports, laboratory studies, past notes, and any other data contained in a clinical sys-tem. As a result, notes that were previously succinct have become unreadable. In contrast, it requires much more ef-fort to summarize the clinically relevant facts. Thus, ironi-cally, health information technology has decreased our abil-ity to manage information. Patients who enter the hospital with hypercalcemia leave with hypercalcemia, and errors are perpetuated verbatim from note to note.

Part of the problem is that trainees are encouraged to document fully to avoid being accused of missing some-thing important and to support billing. However, errors may creep into a long note that is assembled from pieces of other notes. Institutions struggle to develop policies that balance the need to repeatedly document the same information (e.g., often the physical exam does not change from day to day in a hospitalized patient, unless it does) and ensure that important changes are not missed. Currently, there is general agreement that cut and paste or cloning of

Page 7: More Medicine, Fewer Clicks: How Informatics Can Actually ... · mendous resources in developing solutions for better use of cancer data include TriNetX, McKesson Specialty Health,

456 2017 ASCO EDUCATIONAL BOOK | asco.org/edbook

PATT ET AL

clinical notes is undesirable. However, there is not yet a consensus regarding best practice or how to configure clin-ical systems to support best practice. Clearly, the current model where by some reports clinicians are spending twice as much time documenting as they are with patients, is no longer tenable. Creative solutions such as voice-to-text systems with predictive analytic features that can auto- complete notes, scribes that can be present in a clinical en-counter with disrupting workflow or rapport, and structured authoring tools will all need to be exhaustively tested in the field in order to help busy oncologists get through their clin-ical day.

DIGITAL DONATIONS: PATIENT CONSENT AND ENABLING TECHNOLOGIES IN THE EHRUnder the first stages of the meaningful-use EHR incentive program (2010–2015), adoption of EHR systems increased from 51% to 87% in outpatient practices51 and from 16% to 84% in hospitals.52 Increased availability of clinical data (including problem lists, laboratory results, prescription his-tory, and free-text notes) presents a growing opportunity for researchers.53 For example, combining EHR data with adverse event reporting databases has led to automated detection of previously unknown adverse drug reactions.54 EHRs also present an opportunity for prospective research studies, as an adjunct to (and a cross-check for) data collected directly from research participants through traditional paper-based forms or recent innovations using app-based interactions.55

However, researcher access to EHR data has traditionally been limited to institutional settings, where data from a single clinical system or a small network of collaborating systems are available to researchers within the network. These systems expand to form wider networks with more data available to qualified researchers, as in the network of networks established by the Patient-Centered Out-comes Research Institute’s Clinical Data Research Network awardees.56 Such networks are constructed along the grain of institutional boundaries, with careful legal agreements needing to be established among entities as a prerequisite for data sharing. These institution-based studies can pro-vide relatively easy access to EHR data by creating their own legal frameworks for intramural data sharing.

On the other hand, many research studies cut across the grain of institutional boundaries. For example, diseased-fo-cused organizations such as the Multiple Myeloma Research Foundation create community-based registries that identify patients across the country on the basis of disease state and without regard to institutional affiliation.57 We call these participant-based studies. We should highlight that this distinction is not a bright line; studies such as the Precision Medicine Initiative’s All of Us research program pursue a hybrid approach by recruiting from in-network health care systems as well as the general population.58

One model for sharing clinical records with participant- based studies is to engage participants to mediate the trans-fer. For example, after a participant completes an informed

consent process, researchers might ask the patient to col-lect her own clinical records from the hospitals where she has received care. This model confers comprehensive access to clinical records by leveraging a patient’s right under the Health Insurance Portability and Accountability Act to ac-cess a designated record set.59 But the barriers for this model are formidable for otherwise willing participants, including driving to multiple hospitals to visit the medical records departments and filling out multiple authorization forms. In addition, data arriving from faxed or photocopied page-formatted documents instead of electronic structured data add sources of error as well as cost and time.

These shortcomings can be addressed through existing law and additional technology, in particular an application programming interface (API) that can retrieve and move data from EHRs to researchers. Three key enablers are es-tablished by federal laws and regulations: (1) The right un-der the Health Insurance Portability and Accountability Act for a patient to access his or her own medical records, (2) the meaningful-use stage 3 requirement that patients may access their health information with the applications they choose, and (3) the meaningful-use common clini-cal data set, which establishes a minimum set of data to be made available to patients through such an API, includ-ing patient demographics, allergies, immunization, med-ications, laboratory results, and vital signs.60 Of note, the regulations do fall short of defining common standards for the API, which means that certified EHRs could choose to expose these data with proprietary formats and incompat-ible interfaces.

Through a National Institutes of Health–funded and Office of the National Coordinator for Health Information Tech-nology–supported effort called Sync for Science, Dr. Man-del, Mr. Kreda, and colleagues are undertaking a pilot proj-ect with six commercial EHR vendors to establish and test a common, nonproprietary interface for sharing data with research. Building on open standards established through SMART on FHIR and the Argonaut Project,38 Sync for Science has defined a focused set of APIs for EHR vendors to imple-ment. These APIs are published alongside new functionality in each vendor’s patient portal, giving patients the means to approve sharing their data with apps. Under this model, a research study can create an app that asks participants for access to their EHR data. The Sync for Science technology delivers the patient’s approval in the form of an access to-ken following the OAuth 2.0 specification61 (Fig. 3), allowing a research app access to a participant’s EHR data for a des-ignated period of time (typically 1 year).

To support vendor implementation of the Sync for Science APIs, developer documentation (http://syncfor.science/api-calls) and a test suite that connects to each vendor’s portal and provides a compatibility report have been developed. The test suite verifies the availability of sample data, vali-dating that payloads conform to the FHIR specification and checking coded terms against a set of expected vocabular-ies. Although errors and warnings are produced when data fail to match expectations, the tests are permissive, allowing

Page 8: More Medicine, Fewer Clicks: How Informatics Can Actually ... · mendous resources in developing solutions for better use of cancer data include TriNetX, McKesson Specialty Health,

asco.org/edbook | 2017 ASCO EDUCATIONAL BOOK  457

HOW INFORMATICS CAN HELP YOUR PRACTICE

researchers to obtain a richer set of real-world (if occasion-ally messy) data rather than a smaller set of cleaner data. Over the course of 2017, EHR vendors are working to de-ploy this technology at approximately 15 pilot sites around the United States. Although these APIs are designed to sup-port any patient-selected application, the pilot deployment focuses on the Precision Medicine Initiative’s All of Us research program as an initial testbed.

During the pilot phase, a known set of provider sites has been engaged to enable access to a single, well-respected research app, which will provide crucial experience with API-driven data sharing. That said, three impediments in scaling this technology to support a wider ecosystem of re-search studies are anticipated: (1) building a robust share-my-data feature requires a high-quality national provider directory that includes API endpoints for each provider, (2) connecting an app to a provider system still requires reg-istration, a step for which not all vendors have provided an automated approach, and (3) despite regulations that empower patients to access and share their data as they

choose, many health care providers are not yet comfortable enabling connections to unknown apps.

CONCLUSIONMany data systems have evolved to support improved qual-ity of care with greater efficiencies. Despite the richness of available data and the life-threatening nature of cancer, their use throughout oncology practice remains more limited today than in other chronic diseases.

Translating innovations developed in the informatics re-search space into clinical practice is every bit as important as traditional bench-to-bedside translational science. To facilitate such knowledge transfer, ASCO recently launched two journals to explicitly link the cancer and informatics and bioinformatics communities together: JCO Precision Oncology and JCO Clinical Cancer Informatics. Sharing information through these and similar venues, as well as through pre-sentation at conferences, will remain paramount in helping us all benefit from this innovation faster and ultimately al-lowing us to deliver more medicine with fewer clicks.

References

1. Siegel RL, Miller KD, Jemal A. Cancer statistics, 2017. CA Cancer J Clin. 2017;67:7-30.

2. Friedberg MW, Chen PG, Van Busum KR, et al. Factors Affecting Physician Professional Satisfaction and Their Implications for Patient Care, Health Systems, and Health Policy. Santa Monica, CA: Rand Corporation; 2013.

3. Lyman GH, Moses HL. Biomarker tests for molecularly targeted therapies: laying the foundation and fulfilling the dream. J Clin Oncol. 2016;34:2061-2066.

4. National Cancer Institute, Division of Extramural Activities. President’s Cancer Panel—Reports. https://deainfo.nci.nih.gov/advisory/pcp/annualReports/. Accessed January 3, 2017.

5. Murthy VH, Krumholz HM, Gross CP. Participation in cancer clinical trials: race-, sex-, and age-based disparities. JAMA. 2004;291:2720-2726.

6. Tejeda HA, Green SB, Trimble EL, et al. Representation of African-Americans, Hispanics, and whites in National Cancer Institute cancer treatment trials. J Natl Cancer Inst. 1996;88:812-816.

7. Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidence—what is it and what can it tell us? N Engl J Med. 2016;375: 2293-2297.

8. Center for Devices and Radiological Health. Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices: Draft Guidance for Industry and Food and Drug Administration Staff. http://www.

FIGURE 3. Workflow for a Patient Sharing Data With a Research Study, Using an App Built Using Sync for Science Technology

Page 9: More Medicine, Fewer Clicks: How Informatics Can Actually ... · mendous resources in developing solutions for better use of cancer data include TriNetX, McKesson Specialty Health,

458 2017 ASCO EDUCATIONAL BOOK | asco.org/edbook

PATT ET AL

fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm513027.pdf. Accessed January 29, 2017.

9. Sledge GW Jr, Miller RS, Hauser R. CancerLinQ and the future of cancer care. Am Soc Clin Oncol Educ Book. 2013;33:430-434.

10. ASCO forges ahead with CancerLinQ. Cancer Discov. 2014;4:OF4.

11. Patt D. Better screening using big data. J Oncol Pract. 2016;12:699-700.

12. Amarasingham R, Patel PC, Toto K, et al. Allocating scarce resources in real-time to reduce heart failure readmissions: a prospective, controlled study. BMJ Qual Saf. 2013;22:998-1005.

13. Warner JL, Zhang P, Liu J, et al. Classification of hospital acquired complications using temporal clinical information from a large electronic health record. J Biomed Inform. 2016;59:209-217.

14. Antonuzzo A, Vasile E, Sbrana A, et al. Impact of a supportive care service for cancer outpatients: management and reduction of hospitalizations. Preliminary results of an integrated model of care. Support Care Cancer. 2017;25:209-212.

15. Chirgwin JH, Giobbie-Hurder A, Coates AS, et al. Treatment adherence and its impact on disease-free survival in the Breast International Group 1-98 trial of tamoxifen and letrozole, alone and in sequence. J Clin Oncol. 2016;34:2452-2459.

16. Jain P, Keating M, Wierda W, et al. Outcomes of patients with chronic lymphocytic leukemia after discontinuing ibrutinib. Blood. 2015;125:2062-2067.

17. Jain P, Thompson PA, Keating M, et al. Causes of discontinuation and long-term outcomes of patients with cll after discontinuing ibrutinib. https://ash.confex.com/ash/2016/webprogram/Paper95542.html. Accessed January 19, 2017.

18. Dorsey ER, Topol EJ. State of telehealth. N Engl J Med. 2016;375:154-161.

19. Congalton AT, Oakley AM, Rademaker M, et al. Successful melanoma triage by a virtual lesion clinic (teledermatoscopy). J Eur Acad Dermatol Venereol. 2015;29:2423-2428.

20. Sharma M, Marple E, Reichenberg J, et al. Design and characterization of a novel multimodal fiber-optic probe and spectroscopy system for skin cancer applications. Rev Sci Instrum. 2014;85:083101.

21. Marshall CL, Petersen NJ, Naik AD, et al. Implementation of a regional virtual tumor board: a prospective study evaluating feasibility and provider acceptance. Telemed J E Health. 2014;20:705-711.

22. Collins FS, Varmus H. A new initiative on precision medicine. N Engl J Med. 2015;372:793-795.

23. Durack DT. The weight of medical knowledge. N Engl J Med. 1978;298:773-775.

24. Warner JL. Grappling with the data explosion in oncology. Oncol Hematol Rev US. 2015;11:102-103.

25. Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med. 2010;363:501-504.

26. U.S. Food and Drug Administration. Mobile medical applications. http://www.fda.gov/MedicalDevices/DigitalHealth/MobileMedicalApplications/ ucm255978.htm. Accessed January 29, 2017.

27. Shortliffe EH, Scott AC, Bischoff MB, et al. An expert system for oncology protocol management. In Buchanan BG, Shortliffe EH (eds). Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project. Reading, MA: Addison-Wesley; 1984: 653-665.

28. Warner JL, Cowan AJ, Hall AC, et al. HemOnc.org: a collaborative online knowledge platform for oncology professionals. J Oncol Pract. 2015;11:e336-e350.

29. Zon RT, Frame JN, Neuss MN, et al. American Society of Clinical Oncology policy statement on clinical pathways in oncology. J Oncol Pract. 2016;12:261-266.

30. Balmaña J, Digiovanni L, Gaddam P, et al Conflicting interpretation of genetic variants and cancer risk by commercial laboratories as assessed by the Prospective Registry of Multiplex Testing. J Clin Oncol. 2016;34:4071-4078.

31. Ritter DI, Roychowdhury S, Roy A, et al; ClinGen Somatic Cancer Working Group. Somatic cancer variant curation and harmonization through consensus minimum variant level data. Genome Med. 2016;8:117.

32. Swanton C. My Cancer Genome: a unified genomics and clinical trial portal. Lancet Oncol. 2012;13:668-669.

33. Rehm HL, Berg JS, Brooks LD, et al; ClinGen. ClinGen—the Clinical Genome Resource. N Engl J Med. 2015;372:2235-2242.

34. U.S. Food and Drug Administration. Precision Medicine Initiative. http://www.fda.gov/ScienceResearch/SpecialTopics/PrecisionMedicine/default.htm. Accessed January 29, 2017.

35. Li MM, Datto M, Duncavage EJ, et al. Standards and guidelines for the interpretation and reporting of sequence variants in cancer: a joint consensus recommendation of the Association for Molecular Pathology, American Society of Clinical Oncology, and College of American Pathologists. J Mol Diagn. 2017;19:4-23.

36. Warner JL, Jain SK, Levy MA. Integrating cancer genomic data into electronic health records. Genome Med. 2016;8:113.

37. Warner JL, Rioth MJ, Mandl KD, et al. SMART precision cancer medicine: a FHIR-based app to provide genomic information at the point of care. J Am Med Inform Assoc. 2016;23:701-710.

38. Mandel JC, Kreda DA, Mandl KD, Kohane IS, Ramoni RB. SMART on FHIR: a standards-based, interoperable apps platform for electronic health records. J Am Med Inform Assoc. 2016;23:899-908.

39. Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10:523-530.

40. Slight SP, Seger DL, Nanji KC, et al. Are we heeding the warning signs? Examining providers’ overrides of computerized drug-drug interaction alerts in primary care. PLoS One. 2013;8:e85071.

41. Yeh M-L, Chang Y-J, Wang P-Y, et al. Physicians’ responses to computerized drug-drug interaction alerts for outpatients. Comput Methods Programs Biomed. 2013;111:17-25.

42. Mandl KD, Mandel JC, Kohane IS. Driving innovation in health systems through an apps-based information economy. Cell Syst. 2015;1:8-13.

43. Tversky A, Kahneman D. Judgment under uncertainty: heuristics and biases. Science. 1974;185:1124-1131.

44. Clauson KA, Polen HH, Boulos MN, et al. Scope, completeness, and accuracy of drug information in Wikipedia. Ann Pharmacother. 2008;42:1814-1821.

45. Kräenbring J, Monzon Penza T, Gutmann J, et al. Accuracy and completeness of drug information in Wikipedia: a comparison with standard textbooks of pharmacology. PLoS One. 2014;9:e106930.

46. Maskalyk J. Modern medicine comes online: how putting Wikipedia articles through a medical journal’s traditional process can put free, reliable information into as many hands as possible. Open Med. 2014;8:e116-e119.

Page 10: More Medicine, Fewer Clicks: How Informatics Can Actually ... · mendous resources in developing solutions for better use of cancer data include TriNetX, McKesson Specialty Health,

asco.org/edbook | 2017 ASCO EDUCATIONAL BOOK  459

HOW INFORMATICS CAN HELP YOUR PRACTICE

47. Reilly T, Jackson W, Berger V, et al. Accuracy and completeness of drug information in Wikipedia medication monographs. J Am Pharm Assoc. Epub 2016 Nov 17.

48. Rosenbloom ST, Denny JC, Xu H, et al. Data from clinical notes: a perspective on the tension between structure and flexible documentation. J Am Med Inform Assoc. 2011;18:181-186.

49. Yim W-W, Yetisgen M, Harris WP, et al. Natural language processing in oncology: a review. JAMA Oncol. 2016;2:797-804.

50. Middleton B, Bloomrosen M, Dente MA, et al; American Medical Informatics Association. Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA. J Am Med Inform Assoc. 2013;20: e2-e8.

51. Office of the National Coordinator for Health Information Technology. Office-based physician electronic health record adoption. https://dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php. Accessed January 29, 2017.

52. Office of the National Coordinator for Health Information Technology. Non-federal acute care hospital electronic health record adoption. https://dashboard.healthit.gov/quickstats/pages/FIG-Hospital-EHR-Adoption.php. Accessed January 29, 2017.

53. Jensen PB, Jensen LJ, Brunak S. Mining electronic health records: towards better research applications and clinical care. Nat Rev Genet. 2012;13:395-405.

54. Harpaz R, Vilar S, Dumouchel W, et al. Combing signals from spontaneous reports and electronic health records for detection of adverse drug reactions. J Am Med Inform Assoc. 2013;20:413-419.

55. Bot BM, Suver C, Neto EC, et al. The mPower study, Parkinson disease mobile data collected using ResearchKit. Sci Data. 2016;3:160011.

56. Patient-Centered Outcomes Research Institute. Clinical data and patient-powered research networks—awarded projects. http://www.pcori.org/research-results/pcornet-national-patient-centered-clinical-research-network/clinical-data-and-0. Accessed January 29, 2017.

57. Multiple Myeloma Research Foundation. The MMRF CoMMunity Gateway. https://www.themmrf.org/living-with-multiple-myeloma/community-gateway/. Accessed January 29, 2017.

58. National Institutes of Health. All of Us Research Program. https://www.nih.gov/research-training/allofus-research-program. Accessed January 29, 2017.

59. U.S. Department of Health and Human Services. Individuals’ right under HIPAA to access their health information 45 CFR § 164.524. https://www.hhs.gov/hipaa/for-professionals/privacy/guidance/access/index.html. Accessed January 29, 2017.

60. Office of the National Coordinator of Health Information Technology. 2015 edition final rule. https://www.healthit.gov/policy-researchers-implementers/2015-edition-final-rule. Accessed January 29, 2017.

61. Hardt D. The OAuth 2.0 authorization framework. https://tools.ietf.org/html/rfc6749. Accessed January 29, 2017.


Recommended