Evidence Farming1: Implications for Open

ArchitectureIda Sim, MD, PhD

Director, Center for Clinical and Translational Informatics

University of California San FranciscoMay 5, 2011

1With thanks to Rich Kravitz MD, UC Davis and Naihua Duan, Columbia

Rephrasing “Does it Work?”

(Complexes of) Exposures

Outcomestrength of association?

individual

population

Increased breastfeedingText4Baby

Current Approaches: RCT

• Tests prespecified interventions and outcomes• To confirm a hypothesis at the population level• Strong internal validity• Problems: slow to set-up, expensive, short-

term, lack relevance to the real world

ER visits at 1 year

50 people population

100 people

ER visits at 1 year

50 people

Asthma App

Usual Care

Exposures Outcomes?

population

Current Approaches: Data Mining

• Exposures and outcomes from care process systems

• To generate hypotheses at the population level • Problems: limited to data collected, weak internal

validity (data not complete or systematic)

EHR

AppsDec 13, 2009

Guilt

Child care

Worst after school drop-off

AT&T

Current Approaches: N-of-1 Studies

• Within-subject multiple crossover• Only formal method for determining

individual treatment effectiveness• Problems: complicated to set up, analysis

is difficult, little known, not widely used

individual

peak flowpeak flow

Usual Care

Asthma app

Asthma app

Usual Care

Asthma app

Usual Care

Evidence Extraction

• Evidence is something to be extracted from the care process– mining it from the data– directly manipulating the care

process with rigid and pre-defined protocols

Evidence Strip Mining

Evidence Farming

Hay, et al. J Eval Clin Prac 14(2008):707-713.

Rooting for Evidence

Exposures Outcomes?

Industrial Evidence Farming

ER visits at 1 year

50 people population

100 people

ER visits at 1 year

50 people

Asthma App

Usual Care

Personal Evidence Gardens

individual

peak flowpeak flow

Usual Care

Asthma app

Asthma app

Usual Care

Asthma app

Usual Care

Personal Evidence Gardens

individual

dancing

Flovent PRN

Flovent

Flovent

Flovent PRN

Flovent

Flovent PRN

dancing

Crowdsourcing What Matters

• (Complexes of) Exposures– does chocolate trigger (my) asthma?– testing common regimens (ACEI, statin, b-

blocker), complementary medicines

• (Complexes of) Outcomes– what outcomes do patients care about?

Evidence MacrosystemRooting for Evidence

Industrial Evidence Farming

Personal Evidence Gardens

Exposures Outcomes?

individual

dance

Flovent PRN

Flovent

Flovent

Flovent PRN

Flovent

Flovent PRN

dance

ER visits at 1 year

50 people population

100 people

ER visits at 1 year

50 people

Asthma App

Usual Care

How can we scale evaluation?

Stovepiped mHealth

• Health apps built independently– little data sharing

and interoperability

• Limits efficiency and impact of quality mHealth

Internet Hourglass Model

• Standardize and make open the “narrow waist”

• Reduces duplication, spurs community innovation, supports commercial and non-profit uses

OpenmHealth.org

Estrin DE, Sim I. Science; 330: 759-60. 2010.

• The waist should support the evidence macrosystem

OpenmHealth.org

Open Architecture for an Evidence Macrosystem

• Modules for usage analytics– # of text messages, # of sessions, etc.

• Rooting for (glocal) evidence– data sharing with shared syntax and semantics

• Industrial farming, e.g., with RCTs– modules for informed consent, randomization,

adaptive treatment strategy, mixed methods, etc.

• Personal evidence gardening, e.g., N-of-1– modules for scripting and analyzing

individualized N-of-1 protocols, etc.

Open Architecture for an Evidence Macrosystem

• Social media for discovery of exposures and outcomes that matter

• Shared libraries of validated measures and instruments (e.g., PROMIS) – measures that get at finer-grained

mechanisms based on theoretical models of change, etc.

QuickTime™ and a decompressor

are needed to see this picture.

Goal for mHealth Evidence

• A learning community coupled with an open architecture for broad, rapid, and iterative dissemination of evaluation methods and findings that matter

• Ida Sim ida.sim@ucsf.edu• Deborah Estrin

destrin@cs.ucla.edu• http://openmhealth.org/