Deliverable 3.4 Toolkit for Integration of Patient...

FP7-ICT-2013-10 - 5.2 - Virtual Physiological Human

Project No.

610425 Deliverable Report

D3.4, 30/06/2015, Revision: Final Version

Page 1 of 21

RASimAs_D3.4_report 08/07/2015

Deliverable 3.4

Toolkit for Integration of Patient-Specific Data and Physics-Based

Models into VPH Models

Dissemination

Level Type Delivery Month

Confidential (CO)

Restricted (RE)

Public (PU)

Report (R)

Prototype (P)

Other (O)

20

Deliverable D3.4

Milestone Not applicable

Work Package

Leader INRIA

Task/Deliverable

Leader FORTH

Deliverable Due

Date 30/06/2015

Date of Submission

Version 1.7

Keywords

Internal Report Review Done by management body


Project No.



Page 2 of 21


Version Control

Version Date Author (Name, Institution) Comments

1.0

21.05.2015 Stelios Sfakianakis – FORTH-ICS

George Christodoulakis – FORTH-

ICS

1.1 26.05.2015 George Notas – FORTH-ICS

1.2



ICS

1.3 11.06.2015 Júlia Oliveira – UKA-IMI

Thomas Deserno – UKA-IMI

1.4



ICS

George Notas – FORTH-ICS

1.5 24.06.2015 Remi Bessard Duparc – INRIA

1.6

27.06.2015 Konstantinos Marias – FORTH-ICS

Stelios Sfakianakis – FORTH-ICS

George Notas – FORTH-ICS


ICS

1.7 29.06.2015 Júlia Oliveira – UKA-IMI

2.0

3.0

1.X = 1st version circulating between the members / 2.X = 2nd version following

comments of members / 3.X = 3rd final version


Project No.



Page 3 of 21


Table of Contents

LIST OF ACRONYMS .................................................................................................................... 5

1 ABSTRACT .......................................................................................................................... 6

1.1 Context .......................................................................................................6

1.2 Objectives ...................................................................................................6

1.2.1 Deliverable description ............................................................................... 6

2 REQUIREMENTS .................................................................................................................. 7

2.1 Motivation ...................................................................................................7

2.2 Use Case Scenario .....................................................................................8

3 LOGICAL ARCHITECTURE..................................................................................................... 8

4 DISCRETE TOOLKITS ......................................................................................................... 10

4.1 Introduction ............................................................................................... 10

4.2 Anatomical Model Generation Toolkit ....................................................... 10

4.3 Pose Transformation Toolkit ..................................................................... 11

4.4 Volumetric Generation Toolkit .................................................................. 11

4.5 Concluding Remarks ................................................................................ 12

5 TOOLKIT FOR INTEGRATION ............................................................................................... 12

5.1 Introduction ............................................................................................... 12

5.2 Prerequisites ............................................................................................. 13

5.3 Implementation ......................................................................................... 13

5.4 User Interface ........................................................................................... 14

6 CONCLUSIONS .................................................................................................................. 21


Project No.



Page 4 of 21


List of Figures

Figure 1 – Logical Architecture of Data Generation Pipeline of VPH Model ............................ 9

Figure 2 – Workflow for Anatomical Modelling ....................................................................... 10

Figure 3 – “Start-up” Screen ................................................................................................... 14

Figure 4 – “Load Files” functionality ....................................................................................... 15

Figure 5 – “Update File Directories” functionality ................................................................... 15

Figure 6 – “Set Files Directory” functionality .......................................................................... 16

Figure 7 – “VPH Registration” functionality ............................................................................ 16

Figure 8 – “VPH Registration Completed” functionality .......................................................... 17

Figure 9 – “Posing Process” functionality ............................................................................... 18

Figure 10 – “Volumetric Process” functionality ....................................................................... 19

Figure 11- “Single Process” functionality ................................................................................ 20

Figure 12 – “File Selection” functionality ................................................................................ 20


Project No.



Page 5 of 21


List of Acronyms

CGAL The Computational Geometry Algorithms Library

CT Computer Tomography

FORTH Foundation for Research and Technology Hellas

DICOM Digital Imaging and Communications in Medicine

H3D System’s Hardware/Software/Data Integrator by SenseGraphics

INRIA French Institute for Research in Computer Science and Automation

ISS Information Storage System

MRA Magnetic Resonance Angiography

MRI Magnetic Resonance Imaging

RA Regional Anaesthesia

RAAs Regional Anaesthesia Assistant

RASim Regional Anaesthesia Simulator

RASimAs Regional Anaesthesia Simulator and Assistant

ROI Region of Interest

SOFA Simulation Open Framework Architecture

UI User Interface

UKA-IMI University Hospital in Aachen

URJC Rey Juan Carlos University

US Ultrasound

VPH Virtual Physiological Human

VTU ParaView VTK Unstructured Data

X3D XML-based File Format for representing 3D Computer Graphics


Project No.



Page 6 of 21


1 Abstract

This document (Deliverable 3.4) aims at presenting the development of a Java-based desktop

application that will enable the fusion of the distinct modelling toolkits, currently used for the

generation of patient-specific data and physics-based models, into one single platform that will

provide a seamless integration towards the realization of the so called virtual physiological

human (VPH) model. Its main target is the interconnection of those toolkits leading to a fully

automatic procedure. These are the anatomical model, the pose transformation, and the

volumetric generation toolkit.

1.1 Context

Regional Anaesthesia (RA) is an increasingly utilized anaesthesia technique that if properly

performed can have several beneficial effects on patient clinical outcomes. It has also been

suggested that expansion of RA application in Europe could lead to important cost reduction

in health care systems which could reach 100.000€/year and operating theatre. One major

obstacle in the expansion of RA application is the lack of physician training programs leading

to moderate success rates in this subtle technique.

The aim of the RASimAs (Regional Anaesthesia Simulator and Assistant) project is to develop

both a Regional Anaesthesia Simulator (RASim) and a Regional Anaesthesia Assistant

(RAAs) that will ultimately help in the expansion of RA utilization by providing (i) suitable and

quality verified training tools for both novice level and intermediate level anaesthesiologists

and (ii) high tech information technology supported systems that will enable increased success

rates in RA.

Towards the realization of the RASimAs system a crucial step is the creation of a virtual

realistic environment of high fidelity. One of the main components of this environment is a

virtual physiological human model which represents virtually the actual patient under

intervention. A vital requirement for this VPH is to be as close as possible to the actual human

anatomy to provide an unparalleled and reliable training tool for RA.

However, such a VPH is not as simple as someone may think. It is a result of a great deal of

individual and team effort involving numerous hardware and software tools that have to be

linked to each other. Thus the aim of the present document is the description of the

interconnection among well-defined channels leading to the generation of a suitable and

reliable VPH for medical simulation.

1.2 Objectives

1.2.1 Deliverable description

As stated in the Description of Work, the deliverable that constitutes this plan is described as

follows:


Project No.



Page 7 of 21


D3.4 Toolkit for Integration of Patient-Specific Data and Physics-

Based Models into VPH Models

This deliverable is the toolkit for integrating the anatomical and the

mechanical models and individualize each integrated model with patient-

specific data.

Therefore in the present document we describe the design and implementation of the

aforementioned toolkit. The toolkit itself is available as a desktop, cross platform application

that includes the individual model-generating toolkits.

2 Requirements

2.1 Motivation

The different models provide distinct functionalities, focusing in their specific modeling tasks,

while the final outcome should “package” the different outputs in a bundle of artifacts (e.g. files)

suitable for the simulation and assistant environments. In a complex system such as this one,

the toolkit should be as simple as possible describing and implementing the integration of the

individual well-defined stages in the most concise manner. Currently, the generation of all

necessary data of a patient specific VPH for RASimAs is implemented in four distinct stages:

i. Acquisition, preprocessing and archiving of several personalized imaging data (CT,

MRI, MRA, US), organized by parameters such as body region, and patient specific

characteristics such as age, gender, weight, and size.

ii. Enrichment and completion of a patient-specific model by geometrically registering the

different modalities created in the previous stage with respect to the Zygote data or

other VPH model (e.g. Anatomium).

iii. Pose transformation of the VPH models developed at stage (ii).

iv. Generation of a single volumetric mesh of the previously posed VPH model enriched

with information of each anatomical structure such as stiffness.

The first two stages are implemented by partner UKA-IMI, the third one by URJC, while INRIA

is responsible for the fourth task. The whole procedure is currently semi-automatic by

uploading/downloading the relevant output/input data of each distinct stage to/from the

Information Storage System1 (ISS) of RASimAs. Thus, every partner works individually at their

own premises having the ISS as the common link with the rest of the team.

In order to optimize this process all stages should be linked together in an automatic way so

as (i) to simplify and accelerate the process and (ii) to eliminate the burden for the need of a

wide and high level of expertise for the accomplishment of such a complex task. At this point

partner FORTH comes to provide an integrated environment for the orchestration of the

aforementioned distinct stages providing to RASimAs a complete set of personalized data for

medical simulation.

1 Deliverable 2.3 – “Information Storage”


Project No.



Page 8 of 21


2.2 Use Case Scenario

Before proceeding into the architecture design it would be quite helpful to describe the typical

use of the toolkit that provides a unified pipeline of the last three distinct stages for the patient-

specific VPH development.

According to this scenario the user, a domain/modeling expert, initiates the process by

selecting a patient-specific region from the archived personalized image data. At this early

stage the user has to also define the biomechanical parameters of the various anatomical

structures that the specified region comprises. After these initialization steps, the model

integration toolkit takes control and it launches the toolkits for personalized data/Zygote model

registration, for pose transformation of the registered VPH model, and for volumetric mesh

generation in a serial fashion. These processing steps are linked to each other via the

integration toolkit and the final output consists of two files containing the surfaces and

volumetric meshes in X3D and VTU format respectively. These output files are ready then to

be deployed in a training environment for the actual simulation.

3 Logical Architecture

Based on the scenario described in the previous paragraph, Figure 1 depicts the logical

architecture of the VPH model data generation pipeline. The process starts by UKA-IMI that,

after loading the selected personalized image data in DICOM format along with the Zygote

model in X3D format, performs an automatic registration of the chosen data with respect to the

Zygote generalized VPH model, creating in this way a patient-specific VPH model2 which

contains surface meshes in X3D format. The registered VPH model then acts as an input to

URJC which outputs the same model (in X3D format again) after transformation of the various

anatomic structures to a selected pose that is relevant to the actual RA intervention3. Finally,

the pose transformed VPH model is fed into the INRIA toolkit4 for the generation of a single

volumetric mesh in VTU format enriched with important biomechanical parameters used by

the simulation module. The generated posed VPH model in X3D format and the associated

enriched volumetric mesh in VTU format contain all the necessary anatomical and mechanical

data needed by the H3D module developed by SenseGraphics for implementing the simulation

process.

2 Deliverable 3.2 – “Patient-Specific Dataset Library”

3 Deliverable 3.1 – “Toolkit for Pose Transforms of VPH Models”

4 Deliverable 3.3 – “Report on Physics-based models for Body Torso, Upper and Lower Extremities”


Project No.



Page 9 of 21


Figure 1 – Logical Architecture of Data Generation Pipeline of VPH Model


Project No.



Page 10 of 21


4 Discrete Toolkits

4.1 Introduction

As mentioned earlier in this document, currently, the required data processing for turning the

personalized and the Zygote data into the desired simulation format is accomplished in three

distinct stages. Each of these stages has its own toolkit and thus the aim of the current section

is the description of their functionality and technical implementation that will deliver the

indispensable information for the realization of the final integration toolkit.

4.2 Anatomical Model Generation Toolkit

The VPH anatomical model generation toolkit is accomplished by partner UKA-IMI. It is

currently semi-automatic and it comprises five steps according to Deliverable D3.22:

1. Data Selection

2. Gross Registration

3. Data Conversion

4. Fine Registration

5. Data Export

For illustrative purposes Figure 2 below represents the workflow for the anatomical model

generation.

Figure 2 – Workflow for Anatomical Modelling


Project No.



Page 11 of 21


4.3 Pose Transformation Toolkit

The VPH pose transformation is accomplished by partner URJC. This process transforms the

UKA-IMI generated VPH anatomical model to the actual poses that the RA procedure is

performed in real patients. This process is required for several positions of RA (i.e. axilla)

where the MRI data are acquired in a different position from the one used in clinical practice.

It is fully-automatic and it comprises six steps according to Deliverable D3.13:

1. Volumetric Image Building

2. Tetrahedral Mesh Building

3. Weighting

4. Tissue Mapping

5. Tetrahedral Mesh Deformation

6. Tissue Deformation

The first four steps are associated with the pre-processing stage while the last two with the UI

(User Interface) stage. Although the first stage is the most expensive from a computational

point of view, the results showed that the whole process takes less than four minutes to pre-

process the Zygote Body male model. Regarding the second stage, the tetrahedral mesh

deformation and tissue deformation lasted just 14.65ms for the whole Zygote Body model. At

this point it is worthy to mention that the pre-processing stage is the same for any selected

pose, which means that as long as the VPH model is not changed there is no need for this

stage to be repeated.

4.4 Volumetric Generation Toolkit

The VPH volumetric generation toolkit is accomplished by partner INRIA. It is in charge of

transforming the transformed VPH generated by URJC to a single volumetric mesh enriched

with important biomechanical parameters used by the simulation module. It comprises the

following steps:

1. Creation of a Voxel-based image of the anatomy

The surface meshes are imported into SOFA. These meshes are rasterized into voxel-

based images by an engine (specific components that transform input data into output

data), and then these images are merged.

At the end of this step, each voxel of the output image in defined by an integer

describing the type of anatomical tissue which is at its location (e.g. 0 for empty voxels,

1 for bones, 2 for muscles). The image can be saved in a .inr file. It is useful if the

image needs to be reloaded in another scene to apply other processing afterwards (eg.

create a volumetric mesh from this image), as mesh rasterization is not something

instantaneous.

2. Generation of the volumetric (tetrahedral) mesh

For the creation of a tetrahedral mesh from an image, what is used is a component

called MeshGenerationFromImage in SOFA. This component is using the CGAL (The

Computational Geometry Algorithms Library) to create the mesh composed of

tetrahedrons.

For each domain (all voxels that have the same integer value, eg. bones voxels, muscle

voxels, etc.) an algorithm determines an optimized subset of tetrahedra in agreement

with some geometrical parameters (e.g. maximum size and angles of the tetrahedra).


Project No.



Page 12 of 21


The volumetric mesh is exported into a vtk file (.vtu) containing all the descriptions of

the geometrical info.

More information may also be added in this vtk file which is going to be computed

during the mesh generation and that it will just have to be loaded instead of recomputed

every time the needle insertion simulation is launched.

In this particular case, the information about the vertices of the tetrahedral mesh that are in

contact with the bones in the vtu file (these vertices will be fixed in the needle insertion

simulation) has to be saved.

At the end of this process what is generated is a vtu volumetric mesh composed of

tetrahedra describing the different part of the anatomy (the discretization will be useful to

apply different biomechanical parameters for each tissue). This file will be loaded in the

main needle insertion simulation.

4.5 Concluding Remarks

The current section outlined the discrete toolkits that need to be used for the generation of

specific format files for the complete definition of the anatomical and mechanical VPH models

towards the seamless integration with the other components of the RASimAs platform.

However it has to be noticed that, regarding the anatomical model generation toolkit, steps

two, three, and five are not automatic and certain manual adjustments need to be performed

in order to achieve the final goal. Of course this is opposite to the case scenario described

earlier in this document which assumes that the whole procedure is fully automatic, however

it is on the way of getting to a fully automatic procedure. On the other hand the pose

transformation toolkit if fully automatic with overall operation time about four minutes for the

completion of the pre-processing stage which is done only once. Then, as soon as the pre-

processing files are generated the pose transformation takes about 14.65ms. The volumetric

mesh toolkit is fully automatic as well but may need some tuning of the geometrical parameters

during the development of the simulator that will be imperceptible for the final user. The

integration toolkit is nevertheless able to deal with both the automatic and the non-fully

automatic toolkits and allows the user to interact with the individual toolkits during their

execution.

5 Toolkit for Integration

5.1 Introduction

The toolkit for integration is implemented by partner FORTH. Its main target is the

interconnection of the previously discrete toolkits that will lead to a fully automatic procedure.

The integration of the different models is based on a loose coupling data flow paradigm. Each

model is seen as a black box performing its own simulation and processing functionality

whereas the communication is achieved by the exchange of data formatted according to

domain specific formats and standards. At the high level, the objective of this toolkit is to

implement the modeling pipeline, a series of processing steps where the output of each step

is used as input to the subsequent one. There are multiple advantages in this approach:


Project No.



Page 13 of 21


Each processing step is unaware of the existence of the others and therefore there’s a

lot of independency and autonomy.

The lack of inter-dependencies of the processing steps makes the overall system much

more modular. For example the subject posing algorithms can be replaced without

affecting the execution of the processing activities before and after it, as long as the

new module requires the same set of inputs and produces (at least) the same set of

outputs.

The design affords the introduction of more advanced processing to be supported in

the future. For example, some “embarrassingly parallel” processing can be supported

if the total workload of a processing step (e.g. image segmentation) can be split to

multiple independent subtasks that the toolkit can invoke in parallel, assuming that the

underlying host machine offers the processing power needed.

The user experience is significantly improved since the user can interact with a single

application – the toolkit itself – rather than the multitude of the background discrete

simulation modules. The system will be even more user friendly if the background

modules do not require any user interaction after the user provides the relevant

initialization parameter values, since the whole process can be left totally unattended

for as long as the background processing lasts.

5.2 Prerequisites

Towards the implementation of the integration toolkit there are certain prerequisites that have

to be taken into account regarding the functionality of each distinct mechanism that consists

of. These prerequisites are:

1. The common operating system for all distinct toolkits is Windows. The integration toolkit

itself is cross platform but due to the dependencies on the individual toolkits it is in fact

constrained in the common platform.

2. Each toolkit will comprise one executable which is executed as a background,

operating system-level, process in a similar way to console (command-line interface)

applications.

3. The parameters requested for the execution of each toolkit will be passed through the

command line.

4. The inputs of the VPH anatomical model generation toolkit is a single X3D Zygote file

and a series of DICOM MRI images. The output is a single X3D file of the registered

Zygote.

5. The input of the pose transformation toolkit is a single registered Zygote X3D file. The

output is a single posed X3D file.

6. The inputs of the volumetric generation toolkit is a single posed X3D file and specific

biomechanical parameters (e.g. stiffness). The output is a single VTU volumetric mesh

which may enclose the biomechanical parameters.

5.3 Implementation

To accelerate the development and provide a multi-platform software package, the toolkit for

model integration is implemented as a Java-based desktop application. Its design is based on

the notion of a “project” that combines an input data set of DICOM files, a model (e.g. Zygote),

a set of parameters for the execution of the models, and the final results of those models. So


Project No.



Page 14 of 21


the output of a specific “project” is a set of files and other artifacts that can be deployed in a

simulation environment as a new training “scenario”.

After setting up a “project” i.e. providing the input data sets, the parameters, and the Zygote or

an equivalent model, the user is guided through the different processing steps. Each

processing step accepts some input files and produces another set of files. The communication

therefore between the background model-generating software tasks is done through the file

system, and the different programs executing these tasks accept the file names for their inputs.

Of course this design decision introduces some limitations, for example it is assumed that the

tasks run on the same machine. Such an approach though seems the one that requires the

least changes in the software packages of the other partners since all of the individual toolkits

are desktop applications and lack of a network-based interface that would allow their remote

invocation.

The actual execution of each of the applications that implement the processing steps of the

pipeline is implemented as the launch of background (Operating System-level) processes. The

integration toolkit “spawns” each process in turn, providing the set of input files as “command

line” parameters and a temporary directory where the output of each process should be written.

This provides the added benefit that the users can actually interact with every background

application launched, should they want to. Nevertheless the toolkit monitors the execution of

the processing tasks and reports any failure to the user. The availability of the input files of

each step allows the user to re-launch the execution of a failed step or initiate the whole series

of execution steps from the start.

5.4 User Interface

The current section provides a quick walkthrough over the main functionalities of the

Integration Toolkit and the corresponding UI (User Interface) elements.

The initial screen, when the application startups, prompts the user to create a new project

(Figure 3).

As soon as a new project is created the application asks the user to choose the appropriate

file(s) for the implementation of the patient specific VPH (DICOM and commercial VPH

(Zygote/Anatomium)) and Volumetric (Biomechanical parameters) processes (Figure 4).

Figure 3 – “Start-up” Screen


Project No.



Page 15 of 21


Figure 4 – “Load Files” functionality

The available files are loaded from the default locations in the local hard drive (Figure 5), which

allows their update as well. This functionality comes up by navigating into the “File” menu bar

and choosing the “Set files directory” option (Figure 6).

Figure 5 – “Update File Directories” functionality


Project No.



Page 16 of 21


Figure 6 – “Set Files Directory” functionality

As soon as the appropriate files are selected the user may proceed to VPH registration process

(Figure 7).

Figure 7 – “VPH Registration” functionality

After the aformentioned process is completed the application lets the user choose to either

repeat the registration process or navigate backwards/forwards to reselect the files or proceed

to the Posing process (Figure 8).


Project No.



Page 17 of 21


Figure 8 – “VPH Registration Completed” functionality

Towards the completion of the RASimAs data generation procedure next stages should be the

Posing and the Volumetric processes which are represented in Figure 9 and Figure 10

respectively.

At the end of both stages again, the user may repeat the current stage or proceed to either the

previous or the next stage.


Project No.



Page 18 of 21


Figure 9 – “Posing Process” functionality


Project No.



Page 19 of 21


Figure 10 – “Volumetric Process” functionality


Project No.



Page 20 of 21


The whole procedure is implemented in an automatic way by “chaining” all stages one after

the other. Of course, in the case that the user would like to perform either the Posing or the

Volumetric process at a later time without repeating the VPH and the Posing process

respectively, there is the option to perform each of these stages separately by navigating to

the “Process” menu bar and choosing either Posing or Volumetric process (Figure 11).

Figure 11- “Single Process” functionality

Following this, the application prompts the user to select the appropriate VPH or Posing file for

the Posing and Volumetric generation process respectively (Figure 12).

Figure 12 – “File Selection” functionality


Project No.



Page 21 of 21


6 Conclusions

In the current document we have described the toolkit for the integration of separated but well-

defined computational procedures leading to the generation of a suitable and reliable VPH for

medical simulation. In the creation of such a VPH model three independently developed

toolkits are required to be linked to each other in an automatic way so as (i) to simplify and

accelerate the process and (ii) to eliminate the burden for the need of a wide and high level of

expertise for the accomplishment of such a complex task. The first toolkit is dedicated to the

generation of a patient-specific VPH model and particularly to a set of surface meshes of the

different tissues after registering individual MRI data into the Zygote model, the second toolkit

transforms the personalized VPH to the actual poses that the RA procedure is performed in

real patients, while the third one generates a volumetric mesh file enriched with biomechanical

parameters. Then, the generated surfaces meshes and the associated volumetric mesh are

the key components for the execution of the simulation since these two components fully

describe a patient both anatomically and mechanically.

The toolkit for model integration is implemented as a Java-based desktop application that

considers each of the three aforementioned computational stages as a background (Operating

System-level) process. The interface is user-friendly and complete automatic and the only

interaction is the choice of the desired commercial VPH model (e.g. Zygote), patient-specific

DICOM data (e.g. MRI), and specific biomechanical parameters. The current development is

a first version of the Integration Toolkit since all distinct stages, which work fine as stand-alone

platforms, are still under refinement by their developing teams towards their automatic and

seamless integration. Due to this reason the current toolkit makes the use of particular

“dummy” background processes that replace the actual ones, in order to extensively test it for

a reliable and smooth operation. Subsequent versions of the integration toolkit will integrate

newer versions of the individual toolkits as soon as they become available.

Date post:	28-May-2020
Category:	Documents
Upload:	others
View:	13 times
Download:	0 times

Deliverable 3.4 Toolkit for Integration of Patient...

Documents