18 July 2013

Ho-Jin Choi

Dept. of Computer Science, KAIST

Systems Biomedical Informatics Research Center (SBI-NCRC), SNU College of Medicine

Personalized Context-Aware Health Avatar in Smart Phone Environment

The 2013 STI Semantic Summit

Suzdal, Russia, July 17-19, 2013

1

Outline SBI-NCRC – A brief introduction

Research Topics Activity Recognition for Personalized Life-Care

Healthcare Service Framework for Continuous Context Monitoring

Text Mining for Extracting Knowledge from Web Contents

Personalized Bio and Medical Data Analysis

2

Systems Biomedical Informatics National

Core Research Center (SBI-NCRC) - A brief introduction -

3

SBI-NCRC NCRC (National Core Research Center)

Government initiative to support interdisciplinary research & education

Since 2004, one or two centers newly selected each year

Funding scale, 2 million USD/year * 6.5 years

Systems Biomedical Informatics (SBI) Research Center An NCRC established jointly by SNU Hospital and KAIST Computer Science

Born in September 2010

24 professors/researchers participating from 4 organizations SNUH, KAIST, Ajou University, ETRI

Goals for SBI-NCRC To define and realize “Digital Self” or “Health Avatar” prescriptive medicine

To integrate clinical information and bio-information using IT

To launch an interdisciplinary program in Biomedical Informatics

To collaborate towards Joint KAIST-SNUH BIT Campus in Inchon area

4

4P Medicine

Preventive medicine

Predictive medicine

Personalized medicine

Participatory medicine

Tests for early detection

Risk evaluation Prevention Targeted

monitoring

Diagnosis Treatment Results

monitoring

Caring Diseases Caring Health

5

Middleware

Integrative analyses

OCS PACS EMR LIS Seq. Exp. Prot. Tissue CGH

HL

-7

DIC

OM

CD

A

LO

INC

BSM

L

MA

GE

MIA

PE

TM

A

?

SNP

HapMap 2DPage

ProtChip

Tissue

MA

BAC

Chip

Phenomic Self Extractor Genomic Self Extractor

Clinical Genomics

BioData Acquisition

Pattern

Recognizer

DNA

Chip

Transformer

XML Binder

DBConnector

HL-7 / CDA Protocols

Hospital

caBIO

EVS caDSR

CRF CDE/CTEP

caCORE

IDE

Foundation Self

Warehouse

유방암 Application

폐암 Application

혈액암 Application

Legacy System

Authentication/Authorization

Interface

자료 수용기

CGI-

gateway

We

b S

erv

er

XML-Validation

Ontology-Enhancement

Data Indexing

Clinical Trials Knowledge

Base

XML

CGI-gateway

Retrieval

engine

Query

Constructor

Clinical Research

& Clinical Trial KB

Application

Processor

Search engine

Statistical analysis

Visualization

Simulation

Communications

Workflow

Middleware

Ontology Server

Vocabulary Server

Taxonomy Server

Public Bio-DBs

Digital Self

Simulated Self Individuated Second Self

Foundation Self Molecular & Cellular Foundations of Self

Ubiquitous Self Life Logs and Distributed Collaborations

Genomic Self: Translational Bioinformatics

for Genomic Health and Molecular Medicine

Phenomic Self: Data and Measurement driven

Discovery and Understanding of Human Disorders

Physiomic Self: Multi-scale Modeling of Physical and

Physiological Systems of Human Body

Semantic Self: Ontological Representation

and Engineering of Health Avatar

Augmented Self: Multi-modal Assessment and

Treatment to Retain and Enhance Human Performance

Connected Self: Life Logs and Stream-Type Data Mining for Health Protection

Distributed Self: Customized and Context-aware Healthcare Service Agents in Smart Phone Environment

Teaming

7

Group/Project Title PI’s Major

Group 1 Foundation Self: Molecular and Cellular Foundations

of Self SNUH, Ajou U.

Project 1-1 Genomic Self: Translational Bioinformatics for Genomic

Health and Molecular Medicine

Psychiatry(1), Surgery(1),

Bioinformatics(1), Statistics(1)

Project 1-2 Phenomic Self: Data and Measurement driven

Discovery and Understanding of Human Disorders Pathology(2), Bioinformatics(1)

Project 1-3 Physiomic Self: Multi-scale Modeling of Physical and

Physiological Systems of Human Body

Biomedical Engineering(2),

Neurosurgery(1), General

Practice(1)

Group 2 Simulated Self: Individuated Second Self SNUH, KAIST

Project 2-1 Semantic Self: Ontological Representation and

Engineering of Health Avatar

Nursing Informatics(2), Internal

Medicine(1), Pathology(1)

Project 2-2 Augmented Self: Multi-modal Assessment and

Treatment to Retain and Enhance Human Performance

NLP(1), Graphics(1), Image

Processing(1), Psychiatry(1)

Group 3 Ubiquitous Self: Life Logs and Distributed

Collaborations KAIST, ETRI

Project 3-1 Distributed Self: Customized and Context-aware

Healthcare Service Agents in Smart Phone Environment

AI(1), Software Engineering(1),

Bioinformatics(1)

Project 3-2 Connected Self: Life Logs and Stream-Type Data Mining

for Health Protection

Information Systems(1), Data

Mining(1)

Life-style

Logging

Medical

Logging

Genetic

Logging

Virtual Self

Supercomputer

(Collective

Intelligence, Data

Mining)

Healthainment

Smart EMR

SNS Health

Avatars

*

*Autonomous Health Avatar

Virtual Self and Health Avatar

8

9

Target healthcare domains Obesity, diabetes, dementia

On-going research topics Activity Recognition for Personalized Life-Care

(Prof. Ho-Jin Choi)

Healthcare Service Framework for Continuous Context Monitoring (Prof. Jun-Hwa Song)

Text Mining for Extracting Knowledge from Web Contents (Prof. Key-Sun Choi)

Personalized Bio and Medical Data Analysis (Prof. Gwan-Su Yi)

Research Topics

Activity Recognition for Personalized

Life-Care

Prof. Ho-Jin Choi

Dept. of Computer Science

KAIST

10

Multi-Sensor Surveillance for Elderly Care

11 “Patient #1234 is

in a risky situation”

Data observed from microphones helps the system detect the potentially risky situations .

The agent estimates patient #1234’s behaviors.

When preliminary conditions of dangerous situations are occurred to the patient, the agent alarms to the caregiver.

12

Activity Recognition from Video Image with Depth Sensor

Action Recognition with Automatically Detected Essential Body Joints

Technologies Involved

Understand Image Data - RGB images (camera) and depth images (depth sensor) are sent to the system - System then do

- Find a patient in a scene - Track the patient - Understand behaviors of

the patient

★ Issues to challenge - The level of complexity of scenes and behaviors

- Scenes may contain various objects and backgrounds

- Human-behaviors should be understood as much as possible.

13

Understand Audio Data -Audio data (microphones) is sent to the system -System then do

- Detect abnormal sounds

★ Issues to challenge - How accurate the system detects abnormal sounds

Detect Risky Situation -After analyzing data from various sensors, the system determines whether the situation is potentially risky

- System constructs a database for predefined risky situations

- For every situation, the system calculates the likelihood of being risky

- If the likelihood scores more than a threshold, it alarms to the caregiver

★ Issues to challenge -How well the system constructs the database -The accuracy of likelihoods

Find Patient’s Location -Smartphone gives and receives various signals to update patient’s geographic information ★ Issues to challenge - How accurately the system locates the patient

Wrist-Type Device Based Human Behavior Recognition

14

Mediated Interface for human-robot interaction

….

Health Care

Care Services

Raw data (Behavior pattern,

Vital Signal, etc) Old People

How to get “Raw Data” From Old People?

Robots

Care-giver

Activity, Gesture, Vital signal,

Location, Identification (MI: Mediated Interface)

Ex: Watch, Ring

Robots

Care-giver

Elderly Care Services Using Robots

Suggestion

Fall Detection

Wandering Monitoring

Location Monitoring

Care Services

Wrist-Type Device Based Human Behavior Recognition

15

Wrist-type and waist-type monitors

MCU Cortex-M3 (STM32F100)

RF(Zigbee) CC2520

Sensors 3-axis accelerometer(LIS331DLH) 3-axis gyro (L3G4200D) Temperature/humidity (SHT21) Brightness (TCS3414CS) IR Photodiode (TSOP85238)

Emergency button

1개 (front side)

Memory card MicroSD

Battery [Li-Ion 600mAh]

Recharger External rechager

Strap Wrist: nato band Waist: elastic belt

Lifestyle Manager Using SNS and Activity Recognition

16

Life-style patterns

Clinical history

Genetic information

Server

Smartphone

users

Lifestyle

ranking

Life

log

Lifestyle

disease risk

Default

behavior

registration

Location - time

elapse

threshold

Localization by

Wifi signal

17

Analysis of life log and SNS Lifestyle = Eating habit(timing and food types) + CAR(Circadian activity rhythm)

Server

Many smart phone users

Location dimension Sleep : My room, Park, Motel

Rest : TV room, Living room, Lounge

Work or study : Work place, Study place

Enjoy : Shopping place, Cultural place, Attractions

Usual food : Restaurant

Exercise : Exercise place

Religious activities : Church, Buddhist temple

Fast food or snack : Fast food place, Mc. Donald,

Convenience store

Sugar-sweetened beverage : Cafeteria,

Convenience store

Smoking : Smoking place, Convenience store,

Alcohol : Alcohol place, Bar

Drug : Drug store

UNKNOWN

1. Lifestyle

recommendation

2. Measurement of

Lifestyle Metric

Data matrix

Tries for users’ visiting patterns on

location & time dimension

People

Healthcare organizations Proactive/Reactive Services

Lifestyle Manager Using SNS and Activity Recognition

Lifestyle Manager Using SNS and Activity Recognition

18

- Home, Hotel

- Smoking place

- Drinking place

- Working place, Studying place

- Restaurant

- Cafeteria, Coffee shop

- Exercising place

- Religious place

- Attractions, Shopping place,

Cultural place, Enjoy place

- Hospital, Pharmacy

- Unknown

Location manager

- Facebook :

contains more

daily lives than

others

( e.g.,

“I ate a hamburger,

so cool.“ 11:45AM )

Social Network

Services

- Walking

- Exercise / Sport

- Running

- Riding an automobile

- Riding a bicycle

- No activity

Activity Recognition

- Accelerometer

- Illuminance sensor

- WiFi

- GPS

Sensor Handler

- Name

- SNS information

- Address ( GPS )

- BMI

- …

User profile

- Lifelog

- Carlorie counting

(day, week, …)

- Lifestyle disease risk

Service Provider

19

Reuse Experience Base

Developer Community

Social Network Service

Extracted Experiences

(XML-based)

Experience Structure

Problem

Solution

Environments

Flag (success, fail)

Problem Topics

Solution Topics

Document Parser

Social Network Analyzer

Topic Miner

Relationship Creator

Co-related Problem-Solution

Topic Pairs

Constructing case-base Reusing the cases as experience

Experience Mining for Healthcare Social Network

20

Problem Document1

Solution Document1

Original Document1

Original Document2

Original Document3

Original Document4

Original Document10

...

Problem

Document10 . . .

Solution

Document10 . . .

Split to problem and

solution documents

Topic Mining for Problems and Solutions

21

Keywords

window

button

taskbar

bitmap

tiff

hchuldwnd

msdn

hwnd

wparam . . .

Rank Doc #Words Ratio

1 D07 99 80.5%

2 D10 71 67.0%

3 D08 24 24.2%

4 D02 23 29.5%

5 D06 18 13.0%

6 D05 12 13.6%

7 D01 12 14.6%

8 D04 5 27.8%

9 D03 3 8.1%

10 D09 0 0.0%

Problem Topic1 Keywords

http

bit

alpha

wrp

aspx

en

work

system

trusted . . .

Rank Doc #Words Ratio

1 D02 54 69.2%

2 D08 42 42.4%

3 D01 39 47.6%

4 D05 34 38.6%

5 D07 15 12.2%

6 D06 12 8.7%

7 D10 10 9.4%

8 D09 5 71.4%

9 D04 4 22.2%

10 D03 4 10.8%

Problem Topic2 Keywords

windows

vista

microsoft

logo

partner

program

application

support

certification . . .

Rank Doc #Words Ratio

1 D06 108 78.3%

2 D05 42 47.7%

3 D08 33 33.3%

4 D01 31 37.8%

5 D03 30 81.1%

6 D10 25 23.6%

7 D07 9 7.3%

8 D04 9 50.0%

9 D09 2 28.6%

10 D02 1 0.0%

Problem Topic3

Keywords

office

run

microsoft

community

aurigma

test

en

prevent

channel . . .

Rank Doc #Words Ratio

1 D04 20 50.0%

2 D01 17 27.4%

3 D05 15 0.0%

4 D03 9 0.0%

5 D02 7 25.9%

6 D10 0 14.0%

7 D09 0 44.4%

8 D08 0 30.0%

9 D07 0 0.0%

10 D06 0 0.0%

Solution Topic1 Keywords

window

bitmap

net

http

feedback

error

read

link

application . . .

Rank Doc #Words Ratio

1 D05 22 50.0%

2 D01 15 24.2%

3 D03 12 50.0%

4 D02 11 40.7%

5 D04 7 0.0%

6 D10 0 20.6%

7 D09 0 15.6%

8 D08 0 40.0%

9 D07 0 0.0%

10 D06 0 0.0%

Solution Topic2

Keywords

windows

taskbar

system

ws

msft

www

dp

question

bob . . .

Rank Doc #Words Ratio

1 D05 36 50.0%

2 D04 13 50.0%

3 D01 11 17.7%

4 D03 5 50.0%

5 D02 5 18.5%

6 D10 0 33.6%

7 D09 0 28.9%

8 D08 0 16.7%

9 D07 0 50.0%

10 D06 0 50.0%

Solution Topic3 Keywords

button

app

win

msdn

cp

support

creating

usi

bao . . .

Rank Doc #Words Ratio

1 D05 34 0.0%

2 D01 19 40.7%

3 D04 5 0.0%

4 D03 4 0.0%

5 D02 4 14.8%

6 D10 0 31.8%

7 D09 0 11.1%

8 D08 0 13.3%

9 D07 0 50.0%

10 D06 0 50.0%

Solution Topic4

Healthcare Service Framework for

Continuous Context Monitoring

Prof. Jun-Hwa Song

Dept. of Computer Science

KAIST

22

Context-Aware Healthcare Service Scenarios

23

Example Scenarios

Obesity monitoring

Continuously monitors people’s activity level and consumed calories, and suggests proper exercises to the people.

Elderly people monitoring

Continuously monitors an elderly people’s emergency situation such as slipping down on a wet floor, and expedites an emergency call.

Cardiac patient monitoring

Continuously monitors a cardiac patient’s ECG, and expedites an emergency call.

DietSense: Smartphone-Based Diet Monitoring for Enhancing Obesity Self-Care

24

Comparing diet and physical activity

Monitoring Physical Activity Capturing Diet

Calories consumed

from food

Calories burned during

physical activity

camera

microphone accelerometer

Activity Log

Smartphone

1. Collecting activity data from the patient

2. Training ML algorithms for analyzing activity patterns

3. Figuring out the right does time without interrupting

the current work activity

4. Notify the does time and subsequent reminder

Medicine Taker

Motion Sensor

Activity Log

Place A Place A

Task 1 Task 2 Task 3

Action 1-1 Action 1-2

Action 1-3 Action 2-1 Action 2-2

Action 3-1

Type 1

Type 2

Smart Alarming for Long-Term Medicine Adherence

25

Continuous Context Monitoring

26

Continuous monitoring of user’s context A key building block for personal context-aware applications

Often requires complex, multi-step, continuous processing for multiple devices

E.g., Running situation -> sensing in three 3-axis accelerometers, FFT processing, recognition

Location-based ServicesHealthMonitoring

U-Trainer

U-Secretary

U-Reminder

Diet diary

U-Learning

Behavior correction

S

F C

S

S F

F

C

S S

F

C S

S

F

C

S

F

C

Sensing

Feature extraction

Context recognition

PAN-scale dynamic

distributed computing

platform

Context monitoring (e.g., sensing, feature extraction,

recognition)

Application logic

Location-based ServicesHealthMonitoring

U-Trainer

U-Secretary

U-Reminder

Diet diary

U-Learning

Behavior correction

A A

A App logic

Mobile Healthcare Service Framework

27

Develop a healthcare service framework To support multiple and long

running healthcare services with highly scarce and dynamic resources

Efficient resource utilization Longer lasting operation (and

service) under highly scarce resource situation

Quick and efficient abnormal situation detection

Seamless (stable) operation even under high resource dynamics

Challenges Limited battery power due to

mobility Scarce computing resources of

mobile devices Dynamic join/leave of

heterogeneous sensors Multiple healthcare services

share highly limited resources

Resource Manager

Policy Manager

Energy Manager

Sensor Broker

Sensor Detector Communication

Manager

… … Heartbeat monitoring Fall monitoring

Healthcare services

API

Message Interpreter

Sensor Data Processor

Resource Monitor

Network protocols (e.g., ZigBee, BT)

Health Context Monitor

Sensor data

Requests

Sensor availability/status

Results

Sensor detection/control, Data/status report

Sensors in BAN/PAN

Application Broker

Application Interface

Result Manager Message Parser

Diet diary

GPS BVP/GSR Accelerometers … …

Anomaly Detector

Feature Extractor

Resource Coordinator

Healthtopia – Healthcare Platform

Providing API for utilizing health sensors

Saving power consumption for concurrent multiple apps

28

Text Mining for Extracting Knowledge

from Web Contents

Prof. Key-Sun Choi

Dept. of Computer Science

KAIST

29

Food Ingredients and Recipe Advice for Controlling Obesity

Web Environment

Ontology

Mobile Environment

Recipe

Extraction

Web / Wikipedia

Automatic

User Experience

Extraction

Target Food/Dish

Recognition

Manual

User Experience

Input

Web Log / SNS

Equipment

How-to

Food/Dish

Restaurant

DB

Scenario 2.

New Recipe (Low calories)

Suggestion w/ same Ingredients

Scenario 1.

Food-Nutrition Association Visualization

Nutrition

Ingredients

Food-Nutrition

Extraction

30

Mining Connections between Multiple Sources

31

Literature Web Clinical Data

Heterogeneous Textual Sources

- Textbook

- PubMed

- Blogs

- Wikipedia

- Personal health record

Medical information sources

Literature contents affect the Web contents

As background factual knowledge

Web contents have other benefits

Wide coverage

Huge collaborators (confidence)

Aggregating information from multiple sources

Analysis of trend evolving on literature/Web to identify factors that will improve the

quality of patient care

Reliability: Literature > News > Web (Wikipedia, blog, SNS)

Accessibility: Web ≥ News > Literature

Detecting MeSH Keywords from Web Pages

32

Medical Subject Headings (MeSH) NLM controlled vocabulary thesaurus used for indexing articles for PubMed

Tree structure (http://www.nlm.nih.gov/mesh/trees.html)

Provide an efficient way of accessing and organizing biomedical information

Examples of MeSH Headings Body Weight, Kidney, Dental Cavity Preparation, Self Medication, Brain Edema

Extracting Candidates

Matching

Obesity is a medical condition in which excess body fat has a

ccumulated to the extent that it may have an adverse effect on

health, leading to reduced life expectancy and/or increased he

alth problems.[1][2] Body mass index (BMI), a measurement

which compares weight and height, defines people as overweig

ht (pre-obese) if their BMI is between 25 and 30 kg/m2, and o

bese when it is greater than 30 kg/m2.

• Hyperinked terms are extracted as term candidates Language Handling

Detecting MeSH Keywords from Web Pages

33

Extracting Candidates

Matching

Obesity

Medical

condition

Body

fat

Body Mass Index

weight

Dieting

Obesity

Medical

condition

Body

fat

Body Mass Index

Body

Weight

Diet

Link

Structure

MeSH

term

Language Handling

Language Handling Polysemy and homonymy problem

34

Semi-Automatic Infobox Construction for Korean Wikipedia

35

맛있는 감자탕 1그릇을 먹을경우 177Kcal를 섭취하게 된다고 합니다.

Parsing the sentence

Extracting knowledge

<감자탕 1그릇, 열량, 177Kcal>

Infobox DB

Extracting Knowledge from Unstructured Texts using Infobox DB

Personalized Bio and Medical Data Analysis

Prof. Gwan-Su Yi

Dept. of Bio and Brain Engineering

KAIST

36

Personalized Diseases Risk Analysis

User

Agent

Disease risk

Prediction model

Personal genome

Data processing model

Drug response

Prediction model

Disease risk info.

(SNP-Disease)

Drug response info.

(SNP-Drug)

Personal genome

info.

Personal sequence

data

New info. on

disease risk New info. on drug

response

Update Update

request

result

Personalized

disease risk

Genome

profile

Personalized drug

response

Storage

Build database

Personalized Personalized Drug

response Diseases

risk

Obesity, Diabetes Obesity, Diabetes 37

Constructing Databases for Diseases Risk and Drug Response

38

184(Type I Diabetes), 203(Type II Diabetes), 82(Obesity) entries for diseases related SNP markers

collected

228 drugs, 830 SNP markers, 1341entries for drug-SNP related information collected

Diseases

Drugs

Diseases risk info.

SNP ID

Gene

Gene Region (Locus)

Risk Allele

Odds Ratio

P-value

Reference

…

PharmGKB

Drug Bank

Drug response info.

SNP ID

Gene

Gene Region (Locus)

Drug

Condition

Reference

…

Integrated database for

diseases risks and drug responses

Public database

Drug related info.

23andMe

Navigenics

Pathway Genomics

Gene sequencing

service drug related

info.

23andMe

Navigenics

deCODEme

Gene sequencing

service GWAS info.

HugeNavigator

GAD

NCBI (HapMap &

NHGRI catalog)

Public database

GWAS info.

Developing Methods for Analyzing Diseases Risk and Drug Response

39

Diseases

OMIM

PharmGKB

DrugBank

Drugs

PharmGKB

DrugBank

SNPs

dbSNP

HapMap

Genome type

WTCCC

Genome body

UCSC

Ensembl

AceView

Genome

Entrez

Bio. pathway

KEGG

Reactome

NCI pathway

Panther

SNPs

dbSNP

HapMap

Genome type

WTCCC

Biological information

SNP

SNP

Analysis tech. for diseases risk

Analysis tech. for drug response

Obesity, Diabetes

• Extraction of drug response related SNP’s

• Drug targeting and biological pathway based function analysis

• Drug response prediction

• Obesity (Diabetes) related SNP or SNP combinations info.

• Genomic and biological pathway based function analysis

• Diseases risk prediction

Service Platform for Personalized Information about Diseases Risk and Drug Response

Agent

User

Plug-ins from

life-logging team Diseases risk

prediction model

Personal genome info.

data processing model

Drug response

prediction model

Diseases risk info.

(SNP-Disease)

Drug response info.

(SNP-Drug)

Diseases Drugs Genome

body SNP Pathway Genome

type

Life-logging

database

Personal genome

info.

40

Thanks for Attention

Contact:

hojinc@kaist.ac.kr

41