Adopting Site Quality Management to Optimize Risk-Based Monitoring In today’s pressure-packed environment, the quest for improved data quality at a lower cost is of paramount importance to all clinical organizations. A risk-based model for monitoring has become a growing and crucial component in this search. This white paper will break down the issues related to the implementation and tracking of a risk-based monitoring program. WHITE PAPER
Transcript
Adopting Site Quality Management to Optimize Risk-Based Monitoring
In today’s pressure-packed environment, the quest for improved data quality at a lower cost is of paramount importance to all clinical organizations. A risk-based model for monitoring has become a growing and crucial component in this search. This white paper will break down the issues related to the implementation and tracking of a risk-based monitoring program.
WHITE PAPER
Medidata Solutions Worldwide 2
IntroductionAs it is now well documented, clinical R&D organizations have been actively embarking on a paradigm shift in their
approach to ensuring quality in the conduct of clinical trials. In particular, a more targeted, risk-based approach to site
monitoring and data cleaning is being actively pursued by organizations both large and small. The primary goals of this
trial conduct and corresponding data, and the maintenance or even reduction of clinical trial timelines.
following factors:
• Lack of full electronic data capture (EDC) adoption, which severely hampers an organization’s ability to proactively
assess quality across all sites in a clinical trial.
• Concern over the potential of increased regulatory scrutiny, especially in the absence of clear guidance or
endorsement for risk-based approaches from the FDA or EMA.
• Potential for actual decreased trial and data quality, due to less intensive on-site monitoring during trial conduct.
on risk-based quality management in clinical trials. These documents not only provide the industry guidance on how to
demonstrates this:
The following will help you understand the challenges faced when implementing risk-based monitoring and how
developing a site quality management program—especially one leveraging real-time site quality analytics—can simplify
the adoption of this strategy.
1 AUG-2011 FDA Draft Guidance: Guidance for Industry: Oversight of Clinical Investigations — A Risk-Based Approach to Monitoring
“FDA encourages greater reliance on centralized monitoring practices than has been
the case historically, with correspondingly less emphasis on on-site monitoring.”
WHITE PAPER: ADOPTING SITE QUALITY MANAGEMENT TO OPTIMIZE RISK-BASED MONITORING3
The Role of Data Quality and Metrics in the Adoption of Risk-Based Monitoring
one question in this regard: to what extent is patient electronic case report form (eCRF) data actually corrected during
trials, as a result of all of the intense scrutiny traditionally applied to that data by study teams in the form of 100 percent
this, Medidata computed the following two measures using the Medidata Insights™ metrics warehouse, comprising over
2,500 studies from 65 contributing sponsor organizations:
• Total Data Correction Rate: The total percentage of eCRF data found to have one or more updates (“corrections”)
following initial submission of the eCRF form.
• Post-Capture Data Correction Rate: The total percentage of eCRF data found to have one or more updates following
The surprisingly low industry median for these two measures is illustrated in Figure 1. The total data correction rate
comes in at just 4.3 percent, while the post-capture data correction rate is only 2.7 percent. Putting this in context,
reporting, analysis and submission—before any site monitor, data manager or other sponsor representative has had a
chance to scrutinize the data!
It is relevant to note that these metrics currently account for any updates to data values in the EDC audit trail, which
includes natural patient event-based updates, such as adverse event (AE) resolution/outcome information that do not
analysis of AE and concomitant medication forms on various studies reveals a very high “correction” rate on these
percent overall.
0.0%
1.0%
2.0%
3.0%
4.0%
5.0%
Post-CaptureTotal
4.32%
2.68%
FIGURE 1: eCRF DATA CORRECTION RATES
Source: Medidata Insights metrics warehouse
Medidata Solutions Worldwide 4
2 Ibid
clinical trial costs and site monitoring in particular is one of the two largest cost drivers at 30 percent or more. If nothing
approach to ensuring clinical data quality.
The Impact of Regulatory Guidance on Site Management
site monitoring and centralized monitoring. On-site monitoring is the current de-facto practice, so the introduction of
paradigm. But what exactly is centralized monitoring or—more to the point—what should it be? The FDA draft guidance
outlines the primary goals of centralized monitoring that should guide a solution. Among them are the following:
• “Target on-site monitoring by identifying higher risk clinical sites (e.g., sites with data anomalies or a higher frequency
of errors, protocol violations or dropouts relative to other sites)”
• “Augment on-site monitoring by performing monitoring activities that can only be accomplished using centralized
processes (e.g., statistical analyses to identify data trends not easily detected by on-site monitoring)”
• “Conduct aggregate statistical analyses of study data to identify sites that are outliers relative to others and to
evaluate individual subject data for plausibility and completeness”
• “Conduct analyses of site characteristics, performance metrics (e.g., high screen failure rates, high frequency of
eligibility violations, and delays in reporting data), and clinical data to identify trial sites with characteristics correlated
with poor performance or noncompliance”
FIGURE 2: CLINICAL TRIAL COST BREAKDOWN*
IVRS and Drug Distribution
Data Management and Statistics
Project and Clinical LeadershipSite Payments
Monitoring
Other
*Estimated for a large, global clinical trial
Source: Medidata internal analysis
WHITE PAPER: ADOPTING SITE QUALITY MANAGEMENT TO OPTIMIZE RISK-BASED MONITORING5
and other electronic systems. Together, these can help sponsors identify sites with emerging quality-related risks, relative
sites with emerging risks as proactively as possible to remediate any issues, thereby ensuring optimal quality across the
trial. Any implementation of centralized monitoring should therefore include an effective signal detection system that
emphasizes earliest possible detection of emerging quality-related risks at one or more trial sites.
share similar key risk categories including the following:
• Patient safety monitoring and reporting, for example AEs and serious AEs (SAEs),
• Patient data quality, and
• Protocol compliance, including eligibility criteria, and visit and dosing schedules.
Key Metrics to Consider for Adopting Centralized MonitoringAs such, the industry can and should identify a relatively small set of measures that serve as effective surrogates for the
quality of trial conduct by sites in each of these categories. As an example, one might consider the following measures
related to the quality of patient eCRF data capture at each site, all of which should be readily computable using available
EDC-based information:
• Data correction rate (percent of data corrected after initial entry)
• Rate of eCRF queries from clinical data management
•
• Rate of “auto queries” (i.e., queries resulting from programmed data checks within the EDC system)
• Cycle time from patient visit to entry of eCRF data by the site
• Cycle time for sites to respond to queries from the sponsor team
All of the above measures relate to processes that may have a real impact on the quality of eCRF data. One approach
might be to compute all of these measures in parallel to help detect sites with emerging issues. This may have some
degree of success; however, it will more likely lead to a proliferation of signal-detection “noise” that will actually detract
same site process and then choose the one or two that enable the most proactive and effective assessment of site
quality around that process. With respect to data quality, for example, it is fair to say that eCRF query rates (e.g., auto
queries, data management queries, site monitor queries, etc.) and data correction rates should all be good indicators of
the level of quality with which sites are capturing patient data into the sponsor’s EDC system.
Insights metrics warehouse, which reveals a very strong correlation between each of these query rates and data
issues. However, one of them—auto query rate—is clearly the most effective when viewed from the perspective of
early signal detection. This is because, unlike manually generated queries and subsequent data corrections, auto queries
are generated and observable immediately upon entry of the eCRF data at the site. Therefore, both the numerator and
denominator comprising an auto query rate are always current, whereas the numerator in the other rates always lags
behind the denominator by weeks or months (and yes, even years of delay in query generation have been observed!).
Medidata Solutions Worldwide 6
effective among the obvious choices. Note that the Visit-to-eCRF Entry cycle time metric—while it does overlap with auto
query rate as a surrogate for data quality—also measures an additional aspect of site behavior that is worth observing. In
particular, lack of timely attention to capture of patient data in the sponsor’s EDC system—and slowness in responding
to sponsor queries—may portend broader issues with site’s attentiveness to and engagement with the trial overall.
Additionally, various sponsor reviews of incoming EDC data begin losing their value the longer the delay in getting access
to that data. So these cycle times provide an important complement and addition to the auto query rate.
Understanding a Typical Site Quality Management ScenarioIt is indeed important to assess all potential site quality measures in the context of not only how effectively, but how
proactively, they are able to support signal detection. Getting to a relatively small, focused set of standard quality
measures is critical to a successful centralized monitoring implementation. From a tool perspective, organizations and
study teams should be looking for a centralized dashboard in which these quality measures are pre-computed across all
sites and that provides for each site a measure-by-measure and overall assessment of risk-level with respect to quality.
risk levels requiring follow-up. Figure 3 depicts what the tool might display for a given site and measure. In the conceptual
example, the AE rate for the study overall is 8.7 AEs per subject-year, and for site 105 it is 5.6, which puts that site at an
elevated (yellow) risk level for potentially under-reporting AEs.
Note that this would not mean conclusively that the site is under-reporting AEs—only that it is at an elevated risk of such.
The study team would be responsible for deciding what, if anything, is the appropriate course of action to address this
observation. It could be as simple as having the site monitor take some extra time at the next site visit to scrutinize the
site’s AE reporting process and perhaps to review AE reporting requirements with the site staff. It could also include
increasing the amount of SDV targeted for patients enrolled at that site—particularly for the AE forms. Or it could be a
combination of the above. Either way, with a site quality management program, sponsors could better anticipate such
risks and identify the root causes more quickly.
Implementing a Site Quality Management ProgramUsing a standard set of measures, this type of centralized dashboard could be deployed very quickly for each new
study—a critical requirement given the importance of early issue detection during trial conduct. Organizations should also
plan to employ a dashboard tool that is turnkey: not only easy to deploy for each study but also with site quality metrics
automatically assessed and presented in a near real-time fashion. Such a centralized monitoring tool is not only ideal but
quite feasible, and will help your organization revolutionize the way total quality is managed in clinical research.
based monitoring by leveraging the site quality management concepts in this paper, please contact your Medidata
representative.
FIGURE 3: EXAMPLE SITE AND STUDY AE RATE SCORECARD
Site 105 AE Rate
3.7 5.6 6.2 8.7 11.2 13.5
Study AE Rate
Optimizing Clinical Trials: Concept to Conclusion™
BUSINESS ANALYTICS | STUDY & PROTOCOL DESIGN | TRIAL MANAGEMENT, PLANNING & BUDGETING | SITE NEGOTIATION
About Medidata Solutions Worldwide Medidata Solutions is a leading global provider of cloud-based clinical development solutions that enhance the efficiency of customers’
clinical trials. Medidata’s advanced platform lower the total cost of clinical development by optimizing clinical trials from concept to
conclusion: from study and protocol design, trial planning and budgeting, site negotiation, clinical portal, trial management, randomization
and trial supply management, clinical data capture and management, safety events capture, medical coding to business analytics. Our
customers include biopharmaceutical, medical device and diagnostic companies, academic and government institutions, CROs and other
research organizations, encompassing 20 of the top 25 global pharmaceutical companies as well as research organizations of all sizes.
