What is the evidence base for digital
wellbeing ?
Jeremy Wyatt DM FRCP
Professor of eHealth Innovation Professor of eHealth Innovation
Director, Institute of Digital Healthcare, Warwick University
Acknowledgments: Brian McKinstry, Richard Wootton
“Digital wellbeing”
“Improving personal health and wellbeing using
appropriate technologies” [Not about healthy fingers]
• Covers:
– Prevention / health promotion– Prevention / health promotion
– Case finding / screening
– Diagnosis, test ordering
– Managing LTCs
– Improved end of life care…
Why are we asking this question
today?
• Excitement around digital wellbeing
technologies not matched by acceptance /
uptake uptake
• Big industry players active – Phillips, Bosch,
Siemens, Intel…
• Some stakeholders demanding evidence –
genuine concern or stalling tactic ?
Whose point of view ?
User: acceptability, usage rates; satisfaction of people purchasing technology using personal health budget
Clinical: clinical effectiveness (outcomes focus); training and workforce implicationstraining and workforce implications
Technical: feasibility, scalability…
Risk manager: safety, liability
Commissioner : affordability ?
Industry: profitability, market share, risks
Matching the study design to the question
Question Study designs
Acceptability Interviews, focus groups, surveys; usage rates
Usability Speed and accuracy of users with scripted tasks, etc.
Technical feasibility Does it work
Clinical effectiveness Randomised trial with patient relevant outcomes eg.
QOL, mortalityQOL, mortality
Cost effectiveness NICE approach – estimate incremental cost per QALY
using stochastic methods
Affordability Full lifetime cost estimate; revenue release eg. Bed days
Safety Failure mode analysis, modelling, long term large scale
studies
Scalability Modelling, large scale studies
Profitability Balance sheet, share price…
Value for money
“The forthcoming [Scottish] telehealth strategy
must identify opportunities for cost savings from
telehealth delivery across all NHS spending, so
as to ensure the maximum value for money is as to ensure the maximum value for money is
achieved.”
Scottish Government Health and Sport Committee, 3rd Report,
2010 (Session 3)
Case study: York Health Economics 2009
• 18 / 32 Telecare Development Partnerships [56%]
reported avoiding unplanned hospital admissions
• For the 18:
– Estimated reduction in unplanned hospital admissions – Estimated reduction in unplanned hospital admissions
1,220 = 13,870 bed days
– 517 early discharges facilitated by TDP funds, saving 5,668
bed days
– Estimated 518 avoided care home admissions = 61,993 bed
days
• Bottom line – estimated savings £11,151,190 !
YHE study methodsSources of data:
• Quarterly returns from TC development partnerships
• “Partnerships were asked to use local knowledge to estimate the number
of avoidable unplanned admissions that had been prevented during each
quarter due to use of TDP Funds”
• No. of avoided bed days per admission estimated by Partnerships varied
from 1.8 to 29.5… If unable to estimate, YHEC used 4.6 days.from 1.8 to 29.5… If unable to estimate, YHEC used 4.6 days.
What about the 15 Partnerships [44%] not reporting any savings ?
Rejected RCT method as:
• “Partnerships would have to identify a control group with similar
characteristics as TC group”
• “RCT generally unsuitable for evaluating changes in service provision due
to effects of other local and national influences on the objectives”
• “Ethical problems withholding a service with known benefits”
“We know it works”
“The OPALS Major Trauma Study showed that full advanced life-support programs
did not decrease mortality or morbidity for major trauma patients... during
advanced life-support, mortality was greater among patients with Glasgow Coma
Scale scores < 9”
Stiell IG et al. The OPALS Major Trauma Study: impact of advanced life-support on survival and
morbidity. CMAJ. 2008;178:1141-52
Some potential harms from
telehealth
• False positives distract busy clinical staff
• False negatives wrongly reassure patient
• Differential uptake by younger, educated public may worsen health inequalities (“Cyber divide”)worsen health inequalities (“Cyber divide”)
• Some people find TH intrusive or mechanistic - loss of regular human contact
• May expose clinical problems on a scale that NHS cannot manage
• May add to clinical work, take resources from other more effective services
Diabetes telehealth study outcomes
Satisfaction, workload:
• Pts and doctors found systems acceptable in 13/16 studies in Jaana’sSR, pts. in 5/5 studies in Farmer’s SR
• 3 studies reported increased medical workload
• No significant differences in pt. satisfaction or quality of life in 2 RCTs in Welschen’s SRin Welschen’s SR
Pt. adherence: significantly improved in 6/9 RCTs, reported adherence 71%-227% higher in telehealth group (Balas)
Outcomes: 2 pooled estimates showed statistically significant decreases of c. 1.5% in HgbA1c - of doubtful clinical relevance
The argument for systematic reviews
Expert reviews cover a biased sub set of all published studies:
• No expert is up to date - Antman 1994 – experts were 13 years late in recognising the value of clot busters for saving lives from heart attacks
• We all have our biases and selective memory
To get closer to the truth, we must find & synthesise the results of all high quality studies, whatever their results:
• The results of even the best studies will vary randomly around the true effect size
• Negative studies are 2.6 times less likely to be published (Easterbrook 1992]
Negative teledermatology RCT
208 GP dermatology referrals randomised to
usual care or teledermatology; all seen by a
second dermatologistsecond dermatologist
Diagnostic agreement rate: 78% controls, 55%
TD group
IR Bowns et al. Telemedicine in dermatology: a randomised controlled trial. Health
Technology Assessment 2006; Vol. 10: No. 43
Recent SR of home telehealth in COPD
Home telehealth = home telemonitoring and telephone support
Identified 6241 citations… 10 publication covering 858 patients selected for inclusion
4 studies looked at home telemonitoring, 6 randomized trials compared telephone support with usual care.
Results:
• Home telehealth reduced rates of hospitalization and emergency department visits• Home telehealth reduced rates of hospitalization and emergency department visits
• Findings for hospital bed days varied between studies
• Home telehealth interventions were similar or better than usual care for quality of life and patient satisfaction outcomes.
• The mortality rate seemed to be greater in the telephone-support group compared with usual care (risk ratio = 1.2; 95% CI: 0.84 to 1.75).
Polisena et al. Home telehealth for chronic obstructive pulmonary disease: a systematic review and meta-analysis. J Telemed Telecare. 2010;16(3):120-7
Some issues raised
• Included pre-post studies – one with only 18 pts & 12m observation pre, 9m post [missing winter admissions ?]
• QOL – same study said “significant improvement” but gave no data
• Cannot synthesise TH impact on resource usage – 9 different measures used !
• Cannot synthesise TH impact on resource usage – 9 different measures used !
• Mix of interventions & study types – meta-regression needed, but too few studies
• Did not check for publication bias (funnel plot)
• The largest (n=191) of the 6 RCTs was industry sponsored; source of support in 3 others not stated
More rigour in systematic reviews
• Over 100 systematic reviews covering telehealth & telemedicine
• Only 15% of these were eligible for our Medicaid Evidence Based Decisions overview reports 2009
• Half of 15% still fell below acceptable standards of • Half of 15% still fell below acceptable standards of rigour assessed by the AMSTAR reporting quality checklist
Better quality reviews are badly
needed – Cochrane Collaboration
Relevant Cochrane reviews
• Telephone support or telemonitoring for heart
failure, 2008
• Telephone consultation & triage, 2007
• Telephone follow up of discharged pts, 2003• Telephone follow up of discharged pts, 2003
• Telemedicine vs. face to face care, 2000…
Cochrane SR on heart failure TH
25 RCTs: 9 telemonitoring alone, 16 phone support, 2 both up to Nov 2008
Results:
• Telemonitoring saves lives: RR 0.66 (0.54-0.81, p < 0.001)0.001)
• Phone support may not: RR 0.88 (0.76-1.01, p 0.08)
• Both reduced CHF-related admissions RR 0.77 / 0.79
• Several studies showed improved QOL, prescribing, self care, reduced overall costs
Source: Inglis et al, CDSR 2010
Phone versus device based telemonitoring
• 2 RCTs compared simple telephone monitoring of
patients with heart failure by nurses (cost £1200 per
patient year) vs. device-based telemonitoring
(£6000)
• Same impact on admissions & outcomes
• Phone monitoring 5X more cost effective
Chaudhry SI et al. Telemonitoring for patients with chronic heart failure: a systematic
review. Journal of Cardiac Failure 2007; 13:56-62
Cochrane SR on phone consultation / triage
• Range of study types: 5 RCTs, 1 CCT, 3 ITS up to 2007
• Results: 3/5 studies on GP visits showed drop, but 2 showed rise in later visits
• Of 7 studies on A&E usage, 6 showed no change & 1 showed an increaseshowed an increase
• 2 studies looked at deaths – no difference
• “Phone consultation appears to reduce the no. of surgery contacts & OOH visits by GPs, but questions remain on service use, safety, cost & pt. satisfaction”
Burin et al CDSR 2010
Avoiding over optimism
We all welcome evaluations that conform with
our expectations, however poorly conducted,
but…
Conveniently forget to cite (or even publish) Conveniently forget to cite (or even publish)
studies with disappointing results
Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research.
JAMA. 2005;294:218-28.
MSN messenger chat with NHS Direct nurse
Who to carry out & report studies ?
“The perception of a conflict of interest is nearly as important as an
actual conflict, since both erode trust” (World Association of
Medical Editors)
Garg’s systematic review of 100 trials of decision support systems: Garg’s systematic review of 100 trials of decision support systems:
• System developer concluded their own system is effective in
74% of trials
• Independent evaluator concluded that the system is effective in
28% of trials
• 74/28 = 2.6
Garg et al, JAMA 2005
Guest authorship and ghost writing in
rofecoxib studies
• Many trial documents and reviews authored
by unacknowledged industry employees
• “First authorship often attributed to • “First authorship often attributed to
academically affiliated investigators who did
not disclose industry financial support”
Ross et al. Guest authorship and ghost writing in publications
related to rofecoxib: a case study of industry documents. JAMA.
2008;299:1800-12
Reducing the impact of competing interests
• Independent design, conduct and reporting of evaluation studies
• Mandatory declarations of potential competing financial
interests by:– Article authors
– Editors and referees
– Members of grant awarding panels – Members of grant awarding panels
– Those applying for funding
– Members of guideline panels
– Speakers at CPD-approved events
JW competing interests:
• Royalties from Evaluation methods in biomedical informatics textbook
• PhD student bursary for RCT of teledermatology (2001-2005): KSYOS, Amsterdam
• Funds from Scottish Centre for Telehealth 2008/9 to support evaluation
• Hospitality (Spitzbergen 2008): KSYOS, Amsterdam
Suggestions for improving the evidence base
Improve independence of evaluation studies – encourage industry to fund
trials by independent evaluation teams, promote declarations of
competing interest
Use better methods and outcomes - use standard measures of resource
use, validated outcome indicators eg. SF-12, EQ5D; look for harms as
well as benefits
Reduce optimism bias – adopt a healthy scepticism for new techniques
Enhance the skills of policy makers to recognise
rigorous studies – critical appraisal training
Enhance evaluation capacity
Economics
• Wade VA, Karnon J, Elshaug AG, Hiller JE. A systematic review of economic analyses of telehealth services using real time video communication. BMC Health Serv Res. 2010 Aug 10;10:233.
• BACKGROUND: Telehealth is the delivery of health care at a distance, using information and communication technology. The major rationales for its introduction have been to decrease costs, improve efficiency and increase access in health care delivery. This systematic review assesses the economic value of one type of telehealth delivery--synchronous or real time video communication--rather than examining a heterogeneous range of delivery modes as has been the case with previous reviews in this area.
• METHODS: A systematic search was undertaken for economic analyses of the clinical use of telehealth, ending in June 2009. Studies with patient outcome data and a non-telehealth comparator were included. Cost analyses, non-comparative studies and those where patient satisfaction was the only health outcome were excluded.non-comparative studies and those where patient satisfaction was the only health outcome were excluded.
• RESULTS: 36 articles met the inclusion criteria. 22(61%) of the studies found telehealth to be less costly than the non-telehealth alternative, 11(31%) found greater costs and 3 (9%) gave the same or mixed results. 23 of the studies took the perspective of the health services, 12 were societal, and one was from the patient perspective. In three studies of telehealth to rural areas, the health services paid more for telehealth, but due to savings in patient travel, the societal perspective demonstrated cost savings. In regard to health outcomes, 12 (33%) of studies found improved health outcomes, 21 (58%) found outcomes were not significantly different, 2(6%) found that telehealth was less effective, and 1 (3%) found outcomes differed according to patient group. The organisational model of care was more important in determining the value of the service than the clinical discipline, the type of technology, or the date of the study.
• CONCLUSION: Delivery of health services by real time video communication was cost-effective for home care and access to on-call hospital specialists, showed mixed results for rural service delivery, and was not cost-effective for local delivery of services between hospitals and primary care.
• Polisena J, Tran K, Cimon K, Hutton B, McGill S, Palmer K, Scott RE. Home telemonitoring for congestive heart failure: a systematic review and meta-analysis. J Telemed Telecare. 2010;16(2):68-76. Epub 2009 Dec 11.
• We conducted a systematic review of the literature about home telemonitoring compared with usual care. An electronic literature search was conducted to identify studies of home telemonitoring use in congestive heart failure (CHF) patients. Twenty-one original studies on home telemonitoring for patients with patients. Twenty-one original studies on home telemonitoring for patients with CHF were included (3082 patients). A random effects model was used to compute treatment efficacy to measure the average effect of the intervention across all studies where the quantitative pooling of results was appropriate. Home telemonitoring reduced mortality (risk ratio = 0.64; 95% CI: 0.48-0.85) compared with usual care. Several studies suggested that home telemonitoring also helped to lower the number of hospitalizations and the use of other health services. Patient quality of life and satisfaction with home telemonitoring were similar or better than with usual care. More studies of higher methodological quality are required to give more precise information about the potential clinical effectiveness of home telehealth interventions.
Misc SRs
• Jaana M, Paré G, Sicotte C. Home telemonitoring for respiratory conditions: a systematic review. Am J Manag Care. 2009 May;15(5):313-20. Review
• Bartoli L, Zanaboni P, Masella C, Ursini N. Systematic review of telemedicine services for patients affected by chronic obstructive pulmonary disease (COPD). Telemed J E Health. 2009 Nov;15(9):877-83. Review.
• Bolton CE, Waters CS, Peirce S, Elwyn G. Insufficient evidence of benefit: a systematic review of home telemonitoring for COPD. J Eval Clin Pract. 2010 Sep 16. doi: 10.1111/j.1365-2753.2010.01536.x. [Epub ahead of print]
• Neubeck L, Redfern J, Fernandez R, Briffa T, Bauman A, Freedman SB. Telehealth interventions for the secondary prevention of coronary heart disease: a systematic review. Eur J Cardiovasc Prev Rehabil. 2009 Jun;16(3):281-9.
• Maric B, Kaan A, Ignaszewski A, Lear SA. A systematic review of telemonitoring technologies in heart failure. Eur J • Maric B, Kaan A, Ignaszewski A, Lear SA. A systematic review of telemonitoring technologies in heart failure. Eur J Heart Fail. 2009 May;11(5):506-17. Epub 2009 Mar 29
• Inglis SC, Clark RA, McAlister FA, Ball J, Lewinter C, Cullington D, Stewart S, Cleland JG. Structured telephone support or telemonitoring programmes for patients with chronic heart failure. Cochrane Database Syst Rev. 2010 Aug 4;8:CD007228. Review.
• Polisena J, Tran K, Cimon K, Hutton B, McGill S, Palmer K. Home telehealth for diabetes management: a systematic review and meta-analysis. Diabetes Obes Metab. 2009 Oct;11(10):913-30. Epub 2009 Jun 16.
• Dellifraine JL, Dansky KH. Home-based telehealth: a review and meta-analysis. J Telemed Telecare. 2008;14(2):62-6. Review.
• Paré G, Moqadem K, Pineau G, St-Hilaire C. Clinical effects of home telemonitoring in the context of diabetes, asthma, heart failure and hypertension: a systematic review. J Med Internet Res. 2010 Jun 16;12(2):e21. Review.
• Jennett PA, Affleck Hall L, Hailey D, Ohinmaa A, Anderson C, Thomas R, Young B, Lorenzetti D, Scott RE. The socio-economic impact of telehealth: a systematic review. J Telemed Telecare. 2003;9(6):311-20. Review.
Why evaluate health technologies ?
Patients / carers:
• To assure them of safety
• To motivate them to use it
• To answer their own questions (do we know what these are ?)
Clinicians:
• To improve care quality & outcomes, avoid harm• To improve care quality & outcomes, avoid harm
• To help persuade us to use it
Policy makers / service commissioners:
• To inform their purchasing / commissioning decisions
• To help them account for public money
Technology vendors: To market & improve it
Researchers: To build understanding and the evidence base
Some questions about telehealth
How often, & by how much, do these techniques:
• Increase the proportion of the elderly who can be
safely cared for at home ?
• Support self care, improve outcomes ?• Support self care, improve outcomes ?
• Widen access to scarce professional skills ?
• Reduce patient travel & carbon footprint ?
• Decrease healthcare resource utilization, saving
scarce public money ?
What standards of proof for non-drug technologies ?
Randomised trials needed for drugs because:
• There are powerful placebo effects
• Effects can be small and hard to detect
• Drugs are expensive, inconvenient to take
• Spending on new drugs takes investment and attention away from existing effective remedies
• Spending on new drugs takes investment and attention away from existing effective remedies
• Drugs can do harm – some have rare but deadly side effects
But telehealth technologies share most of these features...
Spreading critical appraisal methods
Education in critical appraisal methods using tools from SIGN or Oxford CASP Programme for:
• policy makers
• service commissioners
• medical journalists• medical journalists
• patient groups
Encouraging journalists to publicise independent, well conducted studies that answer useful questions
Who’s involved in telehealth ?
Quality / risk
Technology
provider
Service commissioner
Service manager
Telehealth service Clinical service
Patient Clinician
Quality / risk
manager
TrainerInstallation
and support
Researcher
Service manager
Call handlerInformatics
support
Acceptability of a new technology
10
12
14
16
18
20
Acc
ep
tab
ilit
y o
f te
chn
olo
gy
0
2
4
6
8
10
Scepticism Potential Pilot results Replication Side effects
!
Rational
appraisal
Acc
ep
tab
ilit
y o
f te
chn
olo
gy
Case study 1: Darkins 2008
Claims:
1. Improved outcomes - High level of patent benefit from TH
2. 25% reduction in bed days, 19% reduction in numbers of hospital admissions
3. The cost is $1,600 per patient pa.,” substantially less than other non-
institutional care programs and nursing home care”
Reality:
1. “Outcomes data” = patient satisfaction survey delivered via home
healthcare device every 3m - 60% response rate
2. Compared hospital admissions in 17,025 patients during year pre
enrolment with 6 months post enrolment [improvement could be basis for
allocating patient to CCHT; regression to the mean; seasonal effects - no
simultaneous controls !]
3. Cost comparison – same patients ? What would costs be if no CCHT were
installed ? Huge range of technologies and conditions…
