1

Digital Health: From Online Evidence to Serious Games

“Stop the Bugs”

Patty KostkovaUCL

London, UK

Healthcare at the end of 20th century: Internet

“The impact of the Internet has largely been unforeseen, and it may have a revolutionary role in retooling the trillion-dollar health care industry in the United States” (June Forkner-Dunn 2003).

2

Healthcare in 21st century: mobile 31% (up from 17% in 2010) of cell phone

users have used their phone to look up health information

3

4

Information Overflow

The information you have is not the information you want

The information you want is not the information you need

The information you need is not known

The information that’s known can’t be found in time

5

Projects hosted by the NeLI

portal

National electronic Library of Infection (NeLI, formerly NeLCD) – professionals oriented

Bugs and Drugs on the Web (AR DL) – public oriented

Training in Infection Manual (TII) – for trainees

National Resource for Infection Control (NRIC) – for professionals

eBug – EU DG SANCO project – education pack and games for children about AR

SeaLife – semantic web browser for life sciences

Medicines Support Unit for Optometrists – in collaboration with Department of Optometry, City Uni.

WHO Lab Resources evaluation project

IFH project – International Scientific Forum for Home Hygiene

1. Quantitative evidenceNRIC monthly site usage 2005‐2011

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

Sep‐05

Nov‐05

Jan‐06

Mar‐06

May‐06

Jul‐06

Sep‐06

Nov‐06

Jan‐07

Mar‐07

May‐07

Jul‐07

Sep‐07

Nov‐07

Jan‐08

Mar‐08

May‐08

Jul‐08

Sep‐08

Nov‐08

Jan‐09

Mar‐09

May‐09

Jul‐09

Sep‐09

Nov‐09

Jan‐10

Mar‐10

May‐10

Jul‐10

Sep‐10

Nov‐10

Jan‐11

Mar‐11

May‐11

Page

 Views

Page Views on NRIC before and after ECCMID, 19th - 22nd April 2008

0

100

200

300

400

500

600

700

800

900

1000

Fri 18 Sat 19 Sun 20 Mon 21 Tue 22 Wed 23 Thu 24 Fri 25 Sat 26 Sun 27

Date

Page

Vie

ws

7

Searching/browsing behaviour

8

9

Does the accessibility of information make the search for knowledge any

easier?

10

Google, msn searches that led to the AR site between 1 Sep & 31st Jan

38883 searches 7161 included the word alcohol

12949 antibiotic

6515 chest

3467 bacteria

2085 virus

1852 acne

1149 flu

…. And 250 included "cat"

• Some users have an addiction problem:

• Quit antibiotics

• Quit working

• Many users are concerned about animal

welfare:

• Antibiotics in disadvantaged animals

• information on cat colds

• Others have an international approach:

• swedish acne remedy

• mrsa chinese formula

3 Datasets:

Professional Information Needs: The NeLI/NRIC webserver logs

Public Information Needs: The Google Trends data

Media coverage: Guardian Open Platform API

Professional vs public Needs: Media Influence or Outbreak Early Warning

14

Knowledge and attitude Change: Pre and Post Questionnaires: Bugs and

Drugs Site for Public

Pre and Post Use Questionnaires– Used to evaluate differences in patient knowledge and attitudes

before and after using the site– Users were free to browse the site between questionnaires– First study took place in the Science Museum London as part of

‘live science’– 227 visitors took part of which 177 completed both questionnaires– Study repeated at Nottingham City Hospital Open Day and also

Oxford University Medical School

15

Summary of results - Science Museum

16

Knowledge & Attitude: Bugs and Drugs Relationship between knowledge change and

attitude change

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

-1 -0.5 0 0.5 1

Change in knowledge

Cha

nge

in a

ttitu

de

24% users increased knowledge score and decreased attitude score

7% users decreased knowledge score and decreased attitude score

5% users decreased knowledge score and increased attitude score

11% users increased knowledge score and increased attitude score

Community of Practice – Social Networks

FEM wiki – social networks and training wiki resource for epidemiologists

18

FEM wiki Launch 2010 and beyond

300+ users registered at ESCAIDE event.

Now: over 600 users

A red bar represents a single user’s period of activity.

19

SN vs active forum contributions

SN Roles – evolution in light of actions in the CoP red - 1st replies blue -1st posts and 1st reply green nodes - 1st posters

Evolution of SN

Over 3 milliontweets were collected during May 2009 to

March 2010 containing the word

flu by CeRC pilot study

“I have swine flu”12,954

“I have the flu”12,651

Self Reporting Flu

0

500

1000

1500

2000

2500

3000

20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

Tot

al T

wee

ts

Week

US

UK

Epidemic Intelligence: Early Warning and Outbreak Prediction

11th June 2009

Hour-by-Hour breakdown of the most popular resources posted to Twitter

The Dashboard Platform

Medi+Board

Edugames4allDG SANCO funded project to develop web games teaching pupils about microbes, hand and respiratory hygiene, and appropriate use of antibiotics

Project Background

European wide, DG SANCO funded, antibiotic and hygiene teaching resource for junior and senior school children.

17 countries are involved covering 62% of the European population.

Learning Objectives…

Microbes are all shapesand sizes:

(inside the body)

(in the kitchen)

(on the skin’s surface)

Microbes exist in various locations:

Teaching by playing: Good and Bad Microbes

• Player transports lactobacillus to make yogurtDEMO

Teaching by playing: Food Hygiene

The food sorting game is used to showhand, respiratory and respiratory hygiene.

DEMO

Antibiotics and Vaccine Use

• Player throws white blood cells

• Player delivers full course of antibiotics to infection

• DEMO

Junior Game Evaluation:Statistically significant knowledge gain

Level 3, Q1: “We use good microbes to make things like bread and yogurt” (p < 0.001, chi = 14.46)

Level 1, Q1: “If you cannot see a microbe it is not there” (p = 0.02, chi =5.60)

Level 2, Q2: “Soap can be used to wash away bad bugs” (p =0.02, chi=5.28)

Learning through Mechanics or Text? - using these LO: Statistically significant◦ LO1: “Soap can be used to wash away bad

bugs”◦ LO3: “We use good microbes to make things

like bread and yogurt”

Statistically not significant◦ LO2: “Our bodies have nature defences that

protect us”.◦ LO4: “All microbes and bad for us”.

Demographics 60 children Age: 6-11 years old Recruited on a voluntary basis 30 children in each group Group 1: ◦ Text taught LO1 and LO2◦ Mechanics taught LO3 and LO4

Group 2: ◦ Text taught LO3 and LO4◦ Mechanics taught LO1 and LO2

Results The results do not show a statistical significant for

either method of teaching the LOs – delivering each LO remain subjective to the topic

“We use good microbes to make things like bread and yogurt” was the only statistically significant LO and the one where mechanics was more successful than teaching through text. Also, it was the most successful in the large evaluation using combined text+mechanics approach - > this mechanics works

Work in progress: larger studies, more LOs and combination of methods

Seamless Evaluation Framework

Plot Outline

Quest Set

Conversation

Nodes

Assets

LOs ScoringRules

Story Rules

Game Mechanics

Conversation Rules

Animations

+

+

+

+

+ Presentation Layer

Conversation Layer

Quest Layer

Mission Layer

Education Layer

Study: Integrated assessment vsexternal assessment 49 kids age 6 – 13 Two groups:◦ Integrated assessment: How to be a Millionaire Show?

◦ Non integrated assessment: survey in the game

Results

Game perceived as “fun” (p=0.34, CI=95%) “having fun by playing the game” (p= 0.27,

CI=95%) “nice to play” (p=0.65, CI=95%)

Results continued 63% with integrated evaluation preferred it 20% reported that the integrated evaluation increased

their game experience 30% said game more interesting, another 30% did not

affect them in any way, 15% stated the quiz did not affect their game experience but prefer not to have it integrated in the game

5% do not want the quiz as part of the game 60% did not notice that their knowledge was evaluated

through the integrated evaluation.

IDS Detective Game Interactive Digital Storytelling Game Target group: senior school children (12-16 years) Several Missions◦ Bad Bacteria at BBQ – importance hand hygiene ◦ When Bugs go Wild – prudent antibiotics use◦ Gambling Never Pays - prudent antibiotics use

Microvision…

Investigative dialogue: Jamie Grimesworth (chef)

Collecting EvidenceDEMO PUZZLE

Seamless Evaluation Framework

Plot Outline

Quest Set

Conversation

Nodes

Assets

LOs ScoringRules

Story Rules

Game Mechanics

Conversation Rules

Animations

+

+

+

+

+ Presentation Layer

Conversation Layer

Quest Layer

Mission Layer

Education Layer

Seamless Evaluation Framework

Seamless Evaluation“Debriefing phase”: Post test – towards the

end of the game – measure knowledge change

Case Study

Seamless Evaluation – the EvaluationAims to assess:

Player’s perception of the assessment (21 participants – that did not leave the survey incomplete)

Effectiveness of the game in delivering the LOs (145 participants – participants who finish playing the game)

Participants: A mix of persons playing the game online and in a

controlled environment First the persons played the game and then they were

asked to fill a survey

SE EvaluationPlayers perception of the integrated assessment:

Preferred assessment method: 94% preferred the evaluation integrated in the game flow

Training mission: “Nathan – clean up the locker room!”

Based on “when bugs go wild” mission

Nathan, non-player character, failed to clean the locker room

Dirty socks in footballers’ cocker room

Nathan claims” these are clean but smell off “deodorant”

-> microbision shows presence of microbes

E-Copter lab tests shows these are fungi

Nathan admits the failure and clears the room

Quantifying it: Usability Evaluation

Evaluation in a “London School” 15 kids participated ◦ 8 played with training mission◦ 7 without training mission ◦ Choosing non-control evaluation by selecting only

those who declared that they have never played IDS type of game would demonstrate improvement but we wanted to know “how much” and is it “usable”◦ Small numbers: practical issues in working at schools,

difficulties engaging enough target group children, asking the right questions etc.

Usability Evaluation SUS (system Usability

Scale) adapted to games system -> game use -> play

Results: avg SUS scoreTutorial group = 61.25

(δ=8.95) Non-Tutorial group = 60

(δ=19.94)Not statistically significant p=0.84) considering a

confidence interval of 95%.

# Question p

1 I think I would like to play this game frequently.0.78

2 I found the game unnecessarily complex.0.18

3 I thought the game was easy to play.0.20

4 I think I would need the support of a teacher or otherexpert to be able to play this game.

0.47

5 I found various functions in this game were wellintegrated.

0.13

6 I thought there was too much inconsistency in thisgame.

0.37

7 I would imagine that most people would learn to playthis game very quickly.

0.45

8 I found the game very awkward to use.0.04

9 I felt very confident playing the game.0.13

10 I needed a lot of help before I could get to play thisgame.

0.16

SUS Results Comparison

The only statistically significant difference is for question eight – I found the game awkward to use (p=0.04) for a confidence interval of 95%.

Summary Training mission needed to reduce drop out

rates in children unfamiliar with the IDS games

It worked: 1st evaluation confirmed satisfaction, were not put off, ease of use

2nd evaluation confirmed◦ USABILITY with/without training similar (SUS

score)◦ Kids who played with training found the game less

awkward to use

NOT SURPRISING RESULTS – evidence we achieved our goal

Number of games users per country: Junior Games Senior Game

Number of website visitors, page views and resource downloads per month since January 2008

Media Coverage

Invited Speaker – “Idea Champion” at BMJ Panel: ◦ "The idea most likely to make the biggest impact on healthcare by 2020" (the NHS Innovation

Expo 2011, Excel, London, March 2011)

◦ ECDC Invited Speaker European meeting on “Epidemic Intelligence” (Stockholm, October 2010)

BMJ feature ◦ Can Twitter predict disease outbreaks?

◦ BMJ 2012; 344 doi: 10.1136/bmj.e2353 (Published 17 May 2012

Media Coverage

◦ “Epidemic Inteligence”Scientific Film

◦ BMJ: Medical Innovation: Sabreena Malik

◦ Social networking sites can help prevent pandemics:

(AFP) (15th December 2010) and replicated across more than

30 online news outlets63

“Patty Kostkova and her colleagues at City ehealth Research Centre, City University, London, showed that the 2009 H1N1 flu outbreak could have been identified on Twitter one week before it emerged in official records from general practitioner reports

“E-Bug uses interactive games to teach children about infection control”

CeRC staff – THANKS!CeRCGawesh JawaheerGayo DialloSue WisemanDavid FarrellSteve D’SouzaGemma MadleJane Mani-SaadaAnjana RoyJulius WeinbergMike CatchpoleFaiza HansrajNancy LaiSandy BeverageJohn LawrensonMartin SzomszorLisa LazareckDasun WeerasingheRos NyugiChristina DalyEd de QuinceyHelen Oliver David FowlerSimon Hammond UCL: Stephan Garbin

Justin Moser, Wendy Pan

64

Contact

Thank you to CeRC, ISI Foundation and UCL

p.kostkova@ucl.ac.uk Sites:◦ www.neli.org.uk◦ www.edugames4all.org◦ www.nric.org.uk◦ www.femwiki.com (hosted by ECDC)