Sascha Seifert, Siemens Healthcare, pour la journée e-health 2013

Answers for life. Unrestricted © Siemens AG 2013 All rights reserved.

Theseus Medico and

imaging in the digital diagnosis

Dr. Sascha Seifert

eHealth Day

Sierre

June 2013

Page 2

Unrestricted © Siemens AG 2013 All rights reserved.

Page 2 Dr. Sascha Seifert / Healthcare, Imaging & Therapy, SYNGO

THESEUS-MEDICO consortium

2

Fraunhofer IGD

Siemens CT Erlangen &

Munich

Averbis

Ludwig-Maximillian University Munich

University Hospital Erlangen

Transinsight

DFKI

• 11/2007 – 05/2012

• 67.5 man-years,

• grant 50% government, 50% industry

• 3 research centers, 1 hospital

• 3 companies (Siemens & 2 SME)

• Lead with Siemens AG

Page 3



Codes

Control

1980 - 2000

physics

Robust Learning Methods

Semantic Web Standards

2000-2010

data

Open Internet databases

Business intelligence

Big Data analytics

Semantic interoperability

2010 - future

content

Quelle: Gartner. Hype Cycle for Healthcare Provider Technologies and

Standards, July 2010

Theseus-Medico

(Semantic Web for Medicine)

Google „understands“ now context; knowledge graph with 570 million elements, 18 billion

facts, launched in 2012

RSNA.org drives semantic web

standard for radiology

Evolution of medical data processing

Comprehending Software

Page 4



The THESEUS-MEDICO approach

Semantic Web1 Radiologist / Clinician

• Content Understanding

• Content Linking

• Content Search

• Knowledge explosion

• Internet, books, articles

• Data overload

• Unstructured data: Images, Texts

• Structured data: Lab values, Medication

1by Tim Berners-Lee

https://intranet.app1.healthcare.siemens.com/webcenter/PictureGallery/upload/753/SH_SY_33168_11.jpg

Page 5



Page 6



Application Prototype (National IT-Summit 2011)

2013-06-03 Dr. Sascha Seifert Page 6

Page 7



Text Mining Formalization Text Mining Segmentation Formalization

Sem

anti

c P

roce

ssin

g o

f M

edic

al D

ata

Medical Images Radiology Reports Treatment Plans Online Knowledge Expert Knowledge

Medical Image Annotation

Representational Ontology: OWL

Upper Ontology:

time, space, organization, person, event

mo

re d

om

ain

sp

eci

fic

mo

re li

kely

to

be

ch

an

ge

d

Information Element

Ontology

images, texts,

volumes, …

An

no

tati

on

On

tolo

gy

Clinical Ontology

-doctor, nurse, patient

-medical case

-DICOM Ontology

mid

-leve

l

on

tolo

gy

low

-leve

l

onto

logy

Medical Ontologies

FMA

ICD

-10

We

bsite

Rad

Le

x

FM

A

Ex

ten

.

Ma

pp

ing

s

to e

xte

rna

l

so

urc

es

ICD-10

mapping&

merging

Visual

Charac.

Ma

pp

ing

s

to e

xte

rna

l

so

urc

es

annotation

Thesauri & Taxonomies

extraction

Disease-Symptom Navigation

Multi-Modal Interaction

Quality Control Intelligent Diagnose

Ontology Engineering Information Extraction from Medical Texts

Semantic Search Semantic Reporting Image & Text Linkage

Intelligent Healthcare Applications Intelligent applications using knowledge services

Knowledge Services Infrastructure

Knowledge Extraction and Formalization

Knowledge Management Overview

Clinical Workflow Components

2,5 TB of 750 patients

(≙ ~ 7000 series)

6000 rad reports

Page 8



IMAGE AND TEXT UNDERSTANDING

Page 9



Support systems for disease recognition

Anatomical Body Regions and Organs

Lungs, heart, liver, spleen, kidneys, prostate, urinary bladder,

esophagus, pancreas and several anatomical landmarks

Valve function, coroanry stenosis, osteolytic tumors, liver

tumors, lymph node cancer

Image P

ars

ing

Page 10



Page 10

Increase of knowledge about the image content

Image C

onte

nt

Detection Segmentation

Prior knowledge

# Images 3-fold C.V. [mm] Runtime [s/vol]

Heart* 457 1,30 3,55

Liver 346 1,07 6,00

Spleen 203 2,14 9,90

Right kidney 199 1,03 0,40

Left kidney 197 1,15 0,40

Left lung 166 2,64 1,70

Richt lung 163 2,35 1,80

Urinary bladder 141 1,35 1,00

8 organs, 19 landmarks,

3 body regions <1 min (09/2010)

Generic Image Parsing

Page 11



Knowledge based image processing for anatomy understanding

Mean error 1,80±1,17 mm Mean error 1,70±0,71 mm

Panceas segmentation considering splenic vein

i

Thoracic lymph nodes

Esophagus

Page 12



Osteoblastic Osteolytic

Total Volumes 30 20

Total Annotations 172 42

False Positives per Patient 3.5 3.7

Overall Sensitivity 83% 88%

Mean Sensitivity 80% 93%

Overall Positive Predictive Value 58% 35%

Mean Positive Predictive Value 65% 49%

+9 months Baseline

Th8 osteolytic volume 10% lower end-plate

Th8 mixed

volume 95% whole vertebra Fracture progression

Th11 osteolytic volume 28% vertebra back new

Prior knowledge: vertebrae / discs

Constrains lesions search and automatically find

corresponding lesions in prior exams

Semantic detection and follow-up of spine lesions for bone lesions

Results may vary. Data on file

Page 13



Semantic detection of coronary findings

• Automatic vessel tracing and labeling of coronary

vessels (Random forest)[Gülsün2008]

• Automatic estimation of lumina by Random Forest

Regressor (10x faster than segmentation)

• Characteristics curves along vessel (e.g., degree of

calcification)

• Identification of potential stenosis

• Additional information such as FFR

(measured/simulated) is semantically linked

calcified

non-

calcified mixed overall

by

lesio

n

sensitivity 96.55% 89.23% 91.78% 94.75%

FPR 1.50 2.30 0.87 4.67

by

ve

ssel

sensitivity 98.67% 94.44% 92.16% 96.47%

specificity 79.12% 54.35% 81.39% 71.27%

NPV 99.58% 99.21% 99.30% 99.37%

10-fold cross validation with 256 CCTA volumes runtime <2 min / volume

Knowledge pipeline:

centerlinelumenstenosisclassification

Accelerate and quantify reading Results may vary. Data on file

Page 14



Vessel

MEDICO 14

… use knowledge about the anatomy (shape) of

the liver to register accurately.

Registration of multi modal and multi phase examinations

Instead of just comparing grey-values…

Page 15



MEDICO 15

Registration of multi modal and multi phase examinations

Page 16



MEDICO 16

Fusion view of head-neck tomographies

Segmenting bones Rigid bone registration “Adherence” of soft tissue

Using knowledge about anatomy of bones and soft tissue

Page 17



Semantic Computer Aided Detection

• Horizontal and vertical integration of computer aided detection

• Disadvantage of current solutions:

• Specific sub systems

w/o information exchange /

consolidation

• „Syntactic“ interoperability

Reporting

Semantic Search

Bone lesion CAD

Breast CAD

Liver CAD

Patient context

Bone (e.g. Spine) segmentation

THESEUS-MEDICO:

Standardized, semantic information

using common or (machine-) convertible

vocabulary

Page 18



Three Information Dimensions

of Radiology Reports - Challenges

1 Anatomical / Spatial information

• Location of a finding, e.g. affected organ, lymph nodes

• Spatial modifier, e.g. left, right, axilliary

2 Pathological Information

• Pathological interpretation of the highlighted finding,

e.g. size (enlarged lymph nodes), density (lung nodule),

number of occurrences, etc.

3 Temporal Information

• Provides information about the difference/changes

of the current findings in relation to past findings, e.g.

In comparison to prior examination…

Compound words (German): Stemming

• Split into sub words before semantic mapping to reduce solution space

• Data reduction of 90% in German language

Example:

Unchanged, not pathologically enlarged axilliary,

mediastinal and hilar lymph nodes.

FINDINGS:CHEST (EXAMPLE)

The lungs demonstrate bilateral areas of pulmonary consolidation, involving

predominantly the right upper lobe and to a lesser extent the left upper lobe posterior

lingula and superior segment of the lower lobes with additional patchy opacities in the

right lung base and right middle lobe. Findings are compatible with multi lobar

pneumonia. There are bilateral right greater than left pleural effusions, small in size.

There is diffuse anasarca present with 3rd spacing of fluid. There is an old healed

displaced right clavicle fracture noted. There are sub centimeter hypodense nodules in

both thyroid lobes. There are sub centimeter lymph nodes in the mediastinum measuring

up to 9mmin size in the precarinal space, probably reactive. There are calcified lymph

nodes in the hila bilaterally, from prior granulomatous disease. There is bibasilar passive

atelectasis adjacent to the effusions, with calcified granulomata in both atelectatic lower

lobes. The heart is globally enlarged, with coronary artery, aortic valve calcifications

present. Additional calcified granulomata are shown in the anterior segment of the right

upper lobe. The main pulmonary artery segment is dilated up to 3.3cm in size,

suggesting pulmonary artery hypertension.

Myo|kard|itis

Herz|muskel|entzünd|ung

Inflamm|ation of the heart muscle

muscle

myo

muskel

muscul

inflamm

-itis

inflam

entzünd

KONZEPT

subwort herzheart

card

corazon

card

INFLAMMATIONMUSCLE

HEART

Page 19



100 report texts from lymphoma

cases were manually annotated

RadMiner™ Report Search, Averbis GmbH

Averbis Annotator

(internal)

Validation

Precision=0.921

Recall=0.935

F1=0.928

Orig: Radlex 2.0

Stem: with stemming

Sem: Radlex extended

Semantic Report Search Natural Language Processing + Semantic Mapping

Results may vary. Data on file

Page 20



Knowledge Representation Represent segmentations with semantics

Manuel Möller, DFKI Kaiserslautern, [email protected]

Semantic

Annotation

Medical

Ontology

Link by

Semantic

Concepts

from

Ontology

Page 21



Annotation

Ontology

Consolidated data model: knowledge exchange instead of data exchange using

semantic web standards (RDF and OWL)

new* SPARQL 1.1 (recursive queries)

*W3C Working Draft 05 January 2012

Ontology Terms

Snomed CT 395036

FMA 83281

Radlex 34895

Reference Ontologies

Content

Physical

Ref.

Physical

Ref.

Reference

Ontologies

Representation Language The whole picture

http://en.wikipedia.org/wiki/File:BloodPressure2.jpg

Page 22



SEMANTIC READING

Page 23



Semantic Reading – Compare Studies based on Anatomy instead of

Frame of Reference

Semantics enables registration as expected by radiologists.

Locating corresponding

anatomical structures

by concepts

Anatomy is compared with

the same anatomy

Benefits

•synchronized scrolling

•compare findings over

time

•compare similar patients

bronchial

bifurcation

Page 24



Text-to-Image Linking

(Navigation support)

• Hyperlinks encode semantic information and enable to

bidirectional jumps

• Close semantic gap between image and text based

systems.

• Improve dialogue of radiologist and clinicians (ideal for

radiol. demonstrations, easy access of priors)

Anatomy aware findings labeling

• Benefit from image understanding.

• More meaningful findings names,

understandable by computers

• Enables the system to infer knowledge

Semantic Reading – Intelligent applications

Page 25



Anatomy aware literature references

Image understanding enables to display literature

references matching the underlying anatomy, e.g. the

Bosnial classification for renal cysts. Example subjects:

• Bosniak renal cyst classification

• Fleischner pulmonary nodule management

• Couinaud liver segments

• Lung segments

• Hydronephrosis classification

• TNM

Semantic lesion progression

Related findings retrieved with

semantic reasoning

Semantic Reading – Intelligent applications

Page 26



DISEASE MODELING

Page 27



clinical data

Medical Images

Reports lymph node of head

Lingual lymph node

mandibular lymph node

lymph node

lymph node of trunk

malar lymph node size modifier

shrunken Annotations

facial lymph node

enlarged ME

DIC

O-A

nn

ota

tio

n-O

nto

logy

Wh

at

we a

re u

sin

g

Ou

r e

xte

nsio

n

Use of external knowledge

lymphoma

hasLeadingSymptom

symptom

disease is-a

is-a enlarged lymph node

Disease-Symptom-Ontology

DiSy:hasModifier

DiSy:locatedIn

Infe

r lik

ely

dis

eases

Clinical Recommendations

Next Examinations

From Annotations to Diagnosis

Page 28



Differential

Diagnosis

Definitions

Probabilities

Correlation

Source: Herold, Innere Medizin,

2011

Analysis of available Clinical Knowledge

Page 29



5 Diseases

Hodgkin-Lymphoma

Non-Hodgkin-Lymphoma

Correctal Carcinoma

Reactive Lymphadenitis

40 Symptoms

If possible with definition and

probabilities

Encoding with

RadLex or SNOMED CT

10 Dummy Patients

Information about Leading

Symptoms

Test Data Set provided by experts

Page 30



Ranking factors:

Age, gender, specific incidence

Leading symptoms

Symptom intensity

Reappearing symptoms

Relative importance of symptoms

Ratio of present and absent symptoms

of a disease

Towards a Ranking of Likely Diseases in Terms of Precision and Recall Heiner Oberkampf, Sonja Zillner, Bernhard Bauer, Matthias Hammon, Netmed2012.

Ranking of Likely Diseases

Page 31



Ranked symptoms list is provided

Recommendations of next examinations

Decision Support

Page 32



SEMANTIC SEARCH

Page 33



Medical images can only be searched using:

• meta data in so-called DICOM-headers

(patient name, acquisition date, imaging modality etc.)

• indirectly by searching corresponding radiology reports

‘Content’ of the images can not be used for

• quality control

• data mining for clinical /

epidemiological studies

• decision support

• workflow improvements

• reporting support

Semantic Limitations of Today's Hospital IT

wrt Medical Images

Page 34



Page 35



„Find patients with similar liver lesion, enlarged (pathological) lymph nodes in the thorax, hemoglobin value low and

patient age greater than 65“

Resulting patients

with

similar findings

Reference

lesion

Findings histogram

for query

refinement

Accessing

reports and lab

values

Query

terms

Integrated Semantic Image Search

PET-CT 32%

MRI 12%

Acquisition methods

CHOP14

DXBEAM_C

Treatments

IMVP16

Germany

• DSHNHL 2004-2 (FLYER) Phase 3

• DSHNHL 2006-1B (ACT-2) Phase 3

• DSHNHL 2002-1 (Mega-CHOEP) Phase 3

USA

• UCLA-0406049-01 Phase 3

Clinical trials

Page 36



„Search for images and texts

of patients with thickened wall

of intestine and

hemoglobin value low“

(cancer?)

Including lab values

Understanding the anatomy:

intestine expands to rectum, colon,

sigmoid, cecum, …

Full text search is not enough!

Integrated Semantic Image Search

Page 37



Image Retrieval - Results

• 111 liver lesion with 6105 pairs annotated according to similarity

• Annotated with 5 similarity levels; Leave-One-Out validation

MAP: 0.78

nDCG(10): 0.85

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0.0

0

0.2

5

0.5

0

0.7

5

1.0

0

Recall

Pre

cis

ion

search

inp

ut

output

Hammon, M.; Dankerl, P.; Costa, M.; Tsymbal, A.; Seifert, S.; Sühling, M.;

Uder, M. & Cavallaro, A. (2012), Computer-aided decision support for the

characterization of liver lesions in CT scans, in 'Proceedings of the

European Congress of Radiology (ECR)‘.

Page 38



• transform data into actionable knowledge by image, text and speech understanding

• harmonize CAD by using standardized vocabulary (CAD group, SCR, and 3rd party)

• improve interoperability by describing content in a standardized (semantic web) way

• enable inter-modal semantic navigation, i.e. between image, text and other clinical data

• Understanding content is the key for proactive context-sensitive workflow support

• organize information semantically and prepare for data analytics (understanding first)

• formalize medical knowledge instead of programming

• enable reuse of third party knowledge networks / databases (publishers, clinical trials, ICD10, ….)

Summary

Page 39



Dr. Sascha Seifert

H SY TI / Germany

Hartmannstr. 16

91052 Erlangen

E-mail:

[email protected]

Contact

Disclaimer:

The MEDICO prototypes are under development and not commercially available, and their

future availability cannot be ensured. The prototypes should not be used for any patient

diagnosis or therapy. MEDICO is not related to the commercial hospital information system

Medico.

Acknowledgements:

The MEDICO project is supported in part by the THESEUS program, which is funded by the

German Federal Ministry of Economics and Technology under the grant number

01MQ07016. The responsibility for this demonstration lies with the authors.

mailto:[email protected]

Page 40



syngo. It’s all about you.

Date post:	12-Nov-2014
Category:	Education
Upload:	thearkvalais
View:	827 times
Download:	6 times

Sascha Seifert, Siemens Healthcare, pour la journée e-health 2013

Education