Adopting a Metrics Based Approach for Running Clinical Trials

February 2013

Making trials better, together

Goals9

FOR A METRICS DRIVEN ORGANISATION

Strategic

1 – agreed metrics, supporting company strategy

2 – understand how to control performance

3 – understand cost, quality & time impact of influencing a metric

1 – agreed standard business process

2 – use cases for all metrics with supporting visualisations

3 – defined process variances

Operational

1 – users informed and trained on process and metrics

2 – metrics mapped to process and clinical systems

3 – study specific variances to process managed

Tactical

So, whatdoes achievingthose goals

give us?

Visibility

Insight

Control

That all sounds great…

Steps5

A SIMPLE, PRAGMATIC, PROVEN APPROACH

1ESTABLISH (CURRENT) AND FUTURE STATE PROCESS

Sys

tem

Met

rics

SO

PC

TM

S

Da

taP

roce

ssR

ole

s

3.c.4 [ii] Conduct Interim Visit

CRA

i.5.1

3.c. Site Monitoring3. Site Management and Interim Monitoring

Track and Follow-up on Issues.

Escalate as Necessary 3.c.4.16(i)

Review Supplies, Samples and

Facilities3.c.4.13 New/Current Staff

Training3.c.4.15

CRACRA

ACM

On Site Work Completed for

IMV3.c.4.20

Complete Follow-up Items as NecessaryStatus and Action

Taken

Complete Checklist QuestionsComplete Follow-up Items as

NecessaryLink for any Deviations/

Violations

Complete Checklist Questions

Complete Follow-up Items as Necessary

Link for any Deviations/Violations

If contact not on CTMS

Complete ACM Notification Form

Review Deviations/Violations Against Escalation Criteria

3.c.4.11

Identify Deviations/Violations3.c.4.10

Conduct SDV and Update eCRF and/or CRF Tracking3.c.4.09

CTMS

CRA

EDC via Integration

3.a.4

Complete Checklist QuestionsTrack Site and Subject Deviations/ ViolationsTrack Deviation/Violation Source Data Review

Enter Any Additional Relevant Updates

3.c.4(i)

Liaise with Site Staff as Significant and/or Urgent Issues Arise

3.c.4.12

CRA CRA CRA

Re-evaluate Facilities Including Equip. Calibration

and Temp. Monitoring

3.c.4.14

Visit Completed

Date

CTMS CTMS CTMS CTMSCTMS CTMS

3.c.4.173.c.4.183.c.4.19

2AGREE METRICS AND MAP TO BUSINESS PROCESSThree notes to keep in mind:

Only manage

to what you

can measure If you

measure it,

define how to

fix it

If you

measure

everything,

then you

measure

nothing

2AGREE METRICS AND MAP TO BUSINESS PROCESS

• WHO WILL CONSUME THE METRICS?• Ourselves?• Our customers?• Our shareholders?

• WHAT BEHAVIOURAL AND INFRASTRUCTURE CHANGES WILL BE NEEDED?

• HOW EXACTLY WILL THE MEASUREMENT BE DERIVED?

Considerations as we select the metrics:

2AGREE METRICS AND MAP TO BUSINESS PROCESS

• WHAT WILL WE DO WITH THE RESULTS?• Recovery actions• Thresholds for action

• HOW DO WE PROVE THE DATA IS VALID?• Source / mastering • Timeliness

• ONGOING EVALUATION THAT WE HAVE THE RIGHT METRICS

Considerations as we select the metrics:

Select Metric #

Metric Type

Metric Title Category

21 Q Monitoring Visit Report Completion Monitoring

Formula / Example

Additional Analysis on a "for cause" basisReporting Frequency

Target

Analyze and act on trends during monthly review.

Monthly > 95%

Formula: X/Y (100) = % monitoring visits reports submitted for review within 14 calendar days of site visitX = # final monitoring visit reports submitted for review (= iterations reviewed and subsequently approved) within 14 calendar days of site visitY= total # reports

Example: 5675/6000*100 = 95%Unit of Measure

Compliance with Monitoring Visit Report Completion

Business Driver(s) / Benefit Statement

Definition (see Wiki for detailed definitions)

Compliance with monitoring metrics is directly linked to quality data, patient safety, and GCP (regulatory agency compliance)

Metric Indicator

Reporting Detail

Study / Country / Region / Therapeutic Area / Portfolio levels

LEADING Indicator

%

So, we haveour list……

2AGREE METRICS AND MAP TO BUSINESS PROCESS

Back to the process maps…

Sys

tem

Met

rics

SO

PC

TM

S

Da

taP

roce

ssR

ole

s

3.c.4 [ii] Conduct Interim Visit

CRA

i.5.1

3.c. Site Monitoring3. Site Management and Interim Monitoring

Track and Follow-up on Issues.

Escalate as Necessary 3.c.4.16(i)

Review Supplies, Samples and

Facilities3.c.4.13 New/Current Staff

Training3.c.4.15

CRACRA

ACM

On Site Work Completed for

IMV3.c.4.20

Complete Follow-up Items as NecessaryStatus and Action

Taken

Complete Checklist QuestionsComplete Follow-up Items as

NecessaryLink for any Deviations/

Violations

Complete Checklist Questions

Complete Follow-up Items as Necessary

Link for any Deviations/Violations

If contact not on CTMS

Complete ACM Notification Form

Review Deviations/Violations Against Escalation Criteria

3.c.4.11

Identify Deviations/Violations3.c.4.10

Conduct SDV and Update eCRF and/or CRF Tracking3.c.4.09

CTMS

CRA

EDC via Integration

3.a.4

Complete Checklist QuestionsTrack Site and Subject Deviations/ ViolationsTrack Deviation/Violation Source Data Review

Enter Any Additional Relevant Updates

3.c.4(i)

Liaise with Site Staff as Significant and/or Urgent Issues Arise

3.c.4.12

CRA CRA CRA

Re-evaluate Facilities Including Equip. Calibration

and Temp. Monitoring

3.c.4.14

Visit Completed

Date

CTMS CTMS CTMS CTMSCTMS CTMS

3.c.4.173.c.4.183.c.4.19

MCC21

Sys

tem

Met

rics

SO

PC

TM

S

Dat

aP

roce

ssR

oles

i.5.1 Write Site Visit Report and Review

CTMS

Review Report for Completeness and

Consistencyi.5.1.4

Date of SubmissionName of Reviewer

Report Status = ‘Submitted to Reviewer’

Assign Report to Reviewer

i.5.1.2i.5.2

Generate Completed Visit

Report For Reviewi.5.1.1

i.5. Site Visit Reports

List of Possible Reviewers

CTMS CTMS

CRA

CTMS

CommentsDate of Review

Homepage Alerts for LClinical (Reports Still Due for Review)

3. Site Management and Interim Monitoring

Submit Report for Reviewi.5.1.3

CRA

3.c.4

Unit TrackingNo. of monitors on siteNo. of hours spent on site (by attendee)No of days spent on site by attendee)

4.a.3

2.b.2

2.f.3

CTM/OTL

Report Data

Unit Tracking Data

Report Status = In Progress

CRA

MetricTime from Visit

Completed to Initial Submission

3 BUILD METRIC USE CASES

What metrics are important to us?How do I want to see performance shown?How do I want that performance categorised?

Where are our problem areas?What is the root cause of the problems?What could be influencing the problem?

What do I need to do to resolve issues?What can I do to prevent issues in the future?

4IDENTIFY DATA SOURCES AND DESIGN VISUALISATIONSData and sources:

4IDENTIFY DATA SOURCES AND DESIGN VISUALISATIONSDefine data repository:

• Build for metrics?• Data warehouse?• ETL / web services?

4IDENTIFY DATA SOURCES AND DESIGN VISUALISATIONSBack to the use cases…..

• Tab layout• Graph type• Dimensions• Drill downs• Companion data• Call to action

5IMPLEMENTATION AND CHANGE MANAGEMENT

5IMPLEMENTATION AND CHANGE MANAGEMENT

The Proces

s

The Roles

The How

The Why

What are the key ingredients for change?

31

Dashboard Demo……

Want to know more?

Ben McGraw

By Phone: +1 617 7021602By E-mail:

mds@tibco.com

Duncan Hall

By Phone: +44 1279 812550By E-mail:

duncan.hall@triumphconsultancy.co.uk