Development of a surface-based virtual dental sculpting simulator with multimodal feedback

INTERNATIONAL JOURNAL OF PRECISION ENGINEERING AND MANUFACTURING Vol. 14, No. 4, pp. 577-587 APRIL 2013 / 577

© KSPE and Springer 2013

Development of a Surface-based Virtual DentalSculpting Simulator with Multimodal Feedback

Furqan Ullah1 and Kang Park2,#

1 Graduate School of Mechanical Engineering, Myongji University, San 38-2 Namdong, Cheoin-Gu, Yongin, Gyeonggi-Do, South Korea, 449-728 2 Dept. of Mechanical Engineering, Myongji University, San 38-2 Namdong, Cheoin-Gu, Yongin, Gyeonggi-Do, South Korea, 449-728

# Corresponding Author / E-mail: [email protected], TEL: +82-31-330-6344, FAX: +82-31-321-4959

KEYWORDS: Dental sculpting, Haptic rendering, Mesh subdivision and refinement, Multimodal realities, Virtual reality

This paper presents a surface-based virtual dental sculpting simulator based on sensory modalities like visual, auditory and

haptic sensation. The simulator can be used to perform different dental procedures such as grinding, drilling, or surface

scrubbing, and gain experience of using various virtual dental tools of different shapes. The surface-based dental model, which

is extracted from a commercial 3D dental laser scanner, is used for simulating sculpting processes at less memory cost. Large

amount of triangular mesh data is contained in scanned models; therefore, a model reduction algorithm is proposed for large

triangular mesh data. For the computation of repulsive force feedback, a spring-damper force model with a force filter is used.

Vertex deformation method is implemented along with an enhanced bi-tri subdivision method of triangles to perform precision

sculpting simulation. In order to make the mesh regular, a number of mesh refinement algorithms are performed. Finally,

considering the fidelity, stability, computer efficiency, and update rate of the haptic display, it can be concluded that these

multimodal realities based virtual system can generate stable simulation of material removal from a human tooth model with

realistic auditory, visual, and force sensations.

Manuscript received: October 4, 2012 / Accepted: December 23, 2012

1. Introduction

Medical simulation systems are increasingly being developed and

deployed to teach surgery planning and diagnostic procedures as well

as medical concepts and decision making to personnel in the health

professions. The important aim of the virtual dental simulator is to

provide the pre-operation planning that reduces errors and makes the

dentist feel safer, when entering to perform the real operation task on

the real patient. It also creates new dental training opportunities for

dental treatment procedures. It is believed that during dental surgery

procedures, a dentist must learn to operate dental instruments very

precisely and develop a realistic sense of touch for the interaction of

dental instrument with patient anatomy. Currently, in the dental area,

virtual reality simulators are used for research and development of

dental tools for new therapies, treatments, and early diagnoses.

Different methods were presented to sculpt the triangular models, and

these methods could be applied in medical surgery simulations.1-4

Many researchers have also used haptic display to enhance the

performance of virtual reality environments because simulating touch

in the virtual world can lead to the improvement in the performance of

training simulators. Therefore, many haptic force feedback simulators

based on volumetric and surface data have been proposed for virtual

simulation and visualization of dental training.5-27 Some existing

volumetric and surface based systems for dental simulation are

PerioSim (Haptics-based dental simulator for periodontics),10

Augmented reality haptics system for dental surgical skills training,11

VirDenT (virtual and augmented reality technologies in therapeutic

interventions simulation in fixed prosthodontics),12 HAP-DENT (A VR

Haptic Dental system),13 iDental (A haptics-based dental simulator),14

HapTEL (Haptic Technology Enhanced Learning for dental students).37

In addition, Furqan et al. proposed a surface-based virtual dental surgical

simulator based on axis-aligned dental tools.25 Kim et al. proposed a

haptic dental simulation system based on a hybrid representation of

geometric surface and volumetric representation containing implicit

surface and material properties.15,16 Kiminet al. contributed a new method

to incorporate the distance field of the tool into the dental drilling for

stable feedback force generation as well as accurate volume cutting.17

Wu, J. et al. proposed a volume-based tooth modeling and virtual cutting.

However, most of the systems are still in an exploratory stage, some

important functionalities are still missing in current virtual reality

DOI: 10.1007/s12541-013-0078-y

578 / APRIL 2013 INTERNATIONAL JOURNAL OF PRECISION ENGINEERING AND MANUFACTURING Vol. 14, No. 4

dental simulators.28

Furthermore, in most voxel-based dental training systems, we cannot

use any real tooth model and cannot add a new surgical operation.

Therefore, in this surface-based system, the simulation of material

removal can be carried out on any scanned surface model from a

commercial three-dimensional (3D) dental scanner. Furthermore,

compared with the volumetric-based approach, the surface-based

approach is more complex and difficult to handle but costs lesser

amount of computer memory. On account of the low memory cost, real-

time demands, computational efficiency, accurate visualization, and

smooth haptic feedback associated with surface-based simulators, this

paper focuses on a surface-based virtual dental sculpting simulation with

sensory modalities like visual, auditory and haptic sensation. In this

paper, we propose surface-based haptic dental simulation techniques to

address limitations in previous surface-based systems as follows:

- Triangular subdivision techniques in order to make regular mesh.

- Efficient collision detection with oriented bounding box (OBB)

- Force computation based on spring-damping approach for simulating

realistic tool interaction with the virtual tooth

- Smooth force rendering by using 2ndorder bi-quadratic digital filter.

- Twelve possible cases of bi-tri subdivision method for triangles to

efficiently handle abnormalities associated with the triangle problem

- Simulation of physical contact sound

Sound is an important feature of computer based virtual training

systems and should be included in a realistic simulation. It provides a

strong effect on the emotional perception of a scene. During different

dental operations, realism in simulation can be enhanced significantly

by adding sound effects. The further details of the proposed system are

described below.

For smooth force rendering and cutting simulations in the haptic

environment, the amount of model mesh data must be small. However,

the surface-based model of a human jaw has millions of triangles. Due

to the large amount of data, haptic and visual rendering have some

issues in the real-time updating of the model data. Therefore, in order

to realize stable force feedback, it is better to reduce the amount of data

by efficiently separating of one tooth from the human jaw model. In

this paper, a model reduction algorithm is proposed; by using this

algorithm the desired tooth tissues can be separated from a human jaw

model to achieve a higher visual update frequency of 30-60 Hz and a

stable haptic update rate of 1 kHz. It is also observed that the scanned

model may have irregular triangles that can cause the instability in the

system. Therefore, the mesh triangles must be regular. In order to make

them regular, a number of algorithms have been proposed based on

triangular subdivision techniques.31-33 In real dental treatments, tissues

will be removed according to the interaction between moving tools and

the tissues. This feature involves haptic sensation by computation of

force feedback. For the force feedback, the dental system must have the

ability to efficiently compute the repulsive force and realistic tool

interaction with the virtual tooth, which are all based on efficient

collision detection. In this system, efficient collision is realized using a

bounding box (Bbox) detection method i.e., axis aligned bounding box

(AABB) and OBB. The purpose of the Bbox is to minimize the

computation time required for fast rendering. In addition, many dental

tools with different shapes are used according to different dental

requirements for material removal processes.

In order to perform haptic renderings, a spring-based method is

commonly used for force computations.34-36 The calculation of the

repulsive force depends on the efficient interaction between the virtual

tool and the surface of the tooth. For this purpose, Hooke’s law is used

to calculate the force feedback from deformed mesh. In Hooke’s law

when force is applied to a spring, the spring will be compressed from

one position to another and the displacement during this compression

Fig. 1 Architecture of the virtual system

INTERNATIONAL JOURNAL OF PRECISION ENGINEERING AND MANUFACTURING Vol. 14, No. 4 APRIL 2013 / 579

can be computed. So, the spring force can be computed using this

method with different factors that are involved in the compression.

However, during realizing the repulsive spring force from the deformed

mesh, there are some issues that still need to be resolved such as

instability of the system, low fidelity, and unsmooth haptic sensation.

To handle these problems, a force filter is proposed.

In the surface-based model, the subdivision of triangles during the

sculpting process is an essential factor for simulating smooth material

removal operation. Many researchers introduce novel algorithms to

subdivide the surface and generate interior structures to simulate of the

virtual cutting operation. When the interaction between the virtual

cutter and the tooth surface occurs during the material removal

operation, the size of the mesh triangles must be smaller than the

cutter’s diameter. In order to fulfill this condition, a local-based triangle

subdivision algorithm should be applied to large triangles, and it is also

needed to eliminate the abnormalities of some triangles. In this paper,

we propose twelve possible cases of bi-tri subdivision method for

triangles to efficiently handle abnormalities associated with the triangle

problem. After the deformation of vertices (sculpting process), the

surface tooth model needs to be updated in the real time to enable

correct collision detection in the next computation cycle. In order to

reduce the update time needed for smooth haptic sensation, local

updating of the surface model is performed. Finally, in order to realize

the actual sound during material removal operation, the sound rendering

is performed. Fig. 1 dictates the system architecture of the virtual dental

sculpting system that consists of model reduction, mesh refinement,

collision detection, force computation, haptic and auditory rendering,

local-based subdivision of triangles, and cutting simulation processes.

The rest of this paper is structured as follows. Section 2 describes

the 3D representation of the surface model and different virtual dental

cutters. Section 3 gives the system details i.e., model reduction, mesh

refinement, material removal process, force computation, and

subdivision of triangles, etc. Section 4 explains the experimentsand

results of the proposed system. Section 5 discusses some critical issues

of haptic rendering and Section 6 gives the conclusion.

2. Three-Dimensional Representation

Surface-based modeling costs less computer memory than

volumetric representations. However, it is more complex and difficult

to handle. In this paper, a realistic surface-based 3D human jaw model

is used based on standard geometric references and parameters. The

original jaw model data is extracted from a commercial 3D dental laser

scanner, which defines the boundary surface of the human jaw as

illustrated in Fig. 2(a)-(b). The scanned surface-based data is composed

of a large number of vertices and triangles to construct a realistic virtual

model with the optimum degree of accuracy. The number of 60001

vertices (119995 triangles) is used to visualize the virtual jaw model. In

addition, this simulator is not designed for one specific surface model

but rather the simulation of material removal can be carried out on any

scanned model from a commercial 3D dental scanner.

In the real world, dentists use different kinds of cutter shapes to

sculpture teeth according to several requirements. In this system,

various dental tools with different shapes are included as shown in Fig.

2(c). All dental tools are surface-based modeled with good visual

quality. Dentists can change the size and the shape of a cutter any time

during the real-time sculpting simulation. The ability to change the size

and shape of the virtual dental tool makes the simulator a functional

sculpting system.

3. System Details

3.1 Model Reduction

During the sculpting process of a tooth with haptic interference, a

higher update rate is required for smooth rendering of the haptic

display. However, it is difficult to render a large amount of data with

containing millions of triangles very smoothly because the model must

be updated before the next sculpting action. Due to this problem, an

algorithm is proposed in which the dentist can separate the desired area

of tissues from the original jaw model by creating two cutting planes.

For example, the dentist can separate one tooth from the human jaw by

creating two cutting planes, which define the boundary (i.e., depth and

width) of a separate tooth. The idea is that when two cutting planes

appear in 3D space during the model reduction process, those planes

are set at desired positions that are defined by the boundary of the

tooth, as shown in Fig. 3(a). In the first step, we check which vertices

Fig. 2 Model representation: (a) Surface-based 3D human jaw model

extracted from a commercial 3D dental laser scanner, (b) Wireframe

model of (a), (c) Virtual dental tools of different shapes

Fig. 3 Model reduction: (a) 3D human jaw model with two cutting

planes, (b) After applying the model reduction algorithm. The dark

portion shows the removed tooth area

Fig. 4 Mesh refinement: (a) Original tooth mesh, (b) After applying

two rounds of subdivision, (c) After applying two rounds of Loop

subdivision

3


of the 3D model are inside the two cutting planes by computing the

normal vectors (np1 and np2) of the planes. In the second step, that

vertices are stored into a new data structure and outside vertices are

discarded. With the help of new data structures, a separate tooth requiring

less computer memory is rendered. Fig. 3(b) demonstrates the separation

state of a tooth from the human jaw model with two cutting planes.

3.2 Mesh Refinement

Mostly, the scanned surface-based dental model contains irregular

triangles that may cause the instability in the system during real-time

material removal operation.29,30 In order to realize the smooth tactile

force, it is preferable that the model mesh must contain regular

triangles. A number of algorithms are implemented to make model

mesh regular i.e., , Loop, 3-Split, 4-Split, and 9-Split subdivisions.

The implementation of these algorithms on the virtual tooth model

would be based on the mesh type. However, from our experiments,

the subdivision is recommended for the most dental models

because it produces all triangles after one round of subdivision and

creates C2 (smoothing function in which first and second derivatives

are continuous) surface almost everywhere with the lesser number of

triangles than the Loop subdivision. Note that the mesh refinement

process is implemented before enabling the real-time haptic rendering.

Therefore, there is not any influence of the refinement process on the

real-time material removal rendering. Fig. 5 demonstrates the

implementation of subdivision algorithms on a tooth model that contains

2,888 triangles. After one round of Loop and subdivisions, the

number of triangles was 11,552 and 8,858, respectively. Fig. 4(a) shows

the original tooth mesh and Fig. 4(b) and 4(c) give the results of

and Loop subdivisions, respectively.

3.3 Collision Detection and Sculpting Process

A realistic dental treatment system with haptic interaction requires

natural and real-time interaction between the cutter and the surface of

the tooth. In order to consider the haptic computation frequency of

1 kHz, a Bbox method is used to accelerate the collision detection

between the virtual tool and the surface of the tooth. Bbox method is

adopted to minimize the vertices data computation during the real-time

collision. For the removal of tooth material, a vertex deformation

method is used because of its simplicity and reduced computation

costs. During the collision, the bounding box intersects the surface

before the virtual tool, and when the collision occurs between the tooth

surface and the Bbox, we check how many vertices penetrate the Bbox.

We only consider those penetrated vertices for the second check

between the tool and the penetrated vertex for the deformation

calculation. After that, a new look of the surface can be realized based

on the penetrated vertices. Here, only local triangle properties such as

area, circumcircle radius, and incircle radius of the triangle are

computed to check for abnormalities (too thin/too big: regular/irregular

behaviour) of the triangle. The intersection between the tooth surface

and the Bbox is illustrated in Fig. 5(a), Fig. 5(b) demonstrates the

collision between the spherical tool and the tooth surface, Fig. 5(c)

gives the triangle subdivision, and Fig. 5(d) depicts thedeformed

vertices at the tool’s surface.

3.4 Spherical Collision

In order to detect the collision between a spherical virtual tool and

a mesh vertex, a Bbox is required. In the spherical case, the simplest

and efficient approach is to compute the global space AABB.

Therefore, in order to compute the updating position of the penetrated

vertex by the spherical tool, we simply extend that penetrated vertex to

the sphere’s surface by using following equations.

(1)

(2)

where r is the radius of the sphere, is the distance from the

penetrated vertex to the center of the sphere, Pglobal (Pglobal_x, Pglobal_y,

Pglobal_z) is the position of the penetrated vertex in the global (model)

space, P’global (P’global_x, P’global_y, P’global_z) is the updated position of

the penetrated vertex, Tglobal (Tglobal_x, Tglobal_y, Tglobal_z) is the center of

the sphere.

3.5 Cylindrical Collision

Fig. 6(a) presents the collision between the cylindrical virtual tool

and a mesh vertex. In order to calculate surface coordinates of a

cylindrical virtual tool, the tool can be subdivided into two parts. The

first part is the lower half disk of the sphere and the second part is the

vertical cylinder. The first part is similar to the spherical collision

section. In order to check the collision between the cylindrical part and

the mesh vertex, an OBB is computed in local (cylindrical) coordinate

space. The OBB is a tight fitted dynamic bounding box containing the

cylindrical part of the virtual tool. We can simplify the collision

computation by converting the eight corners of the OBB into the global

3

3

3

3

P'global r dΔ dΔ⁄( ) Tglobal+= 0 dΔ r< <

dΔ Pglobal Tglobal–=

dΔ

Fig. 5 Collision detection: (a) Intersection between the virtual Bbox and the tooth surface, (b) Intersection between the cutting tool and the tooth

surface, (c) Occurrence of subdivision, (d) Updated position of penetrated vertices


(model) coordinate space AABB. After that, the collision is checked by

computing the maximum and minimum of the AABB. In order to

update the position of the penetrated vertex to the surface of the

cylinder, the inverse transformation (rotation and translation) is applied

to transform the vertex position from global space to the local space.

For this purpose, following operation is used to transform the global

vertex into the local space.

(3)

where Plocal (Plocal_x, Plocal_y, Plocal_z) is the vertex position in the local

space and M-1 is the inverse of the 4 × 4 homogenous transformation

matrix. Thus, the surface coordinates P’local of the cylinder can be

calculated as follows:

(4)

(5)

(6)

where r is the radius of the cylinder, θ is the angle between the center

axis and the outer edge of the cylinder and Δd’ is the distant from the

penetrated vertex position to the center axis of the cylinder. Finally, the

vertex is updated at the surface of the cylinder, and then the

transformation is performed again from the local space to the global

space by using the following relation:

(7)

3.6 Cone Collision

A cone can also be subdivided into two parts: a vertical cylinder and

a cone. The collision between the cone and the mesh vertex can also

be checked by subsequently applying the same procedure of the

cylindrical collision. After performing the inverse transformation by

using Eq. 3, the surface coordinate P’global can be calculated as follows:

(8)

where r is the radius of the bottom circle of the cone, hc is the height

of the cone, and hp is the distance from the origin of the cone to a

penetrated vertex along the z-axis. After computing surface coordinates

of the cone, the transformation is needed to be performed again from

the local coordinate system to the global coordinate system by using

Eq. 7. The collision between the virtual cone tool and a mesh vertex is

demonstrated in Fig. 6(b).

3.7 Haptic Rendering

Generally, graphic applications in real-time virtual environment

have display update requirements of 30-60 frames per seconds. The

update rate of haptic renderings must be higher than 1 kHz in order to

maintain a stable force feedback system.25-30 In order to realize the

force feedback in haptic simulations, the spring-damping force model

is commonly used to transform the motion of the haptic tip to the

virtual force. Therefore, the spring-damping approach is adopted to

compute the force feedback. The general force vector equation can be

written as:

(9)

where Ks and Kd are the spring stiffness and damping coefficient

respectively, is the displacement of the spring, and is the velocity

of the virtual tool after collision. For smooth and stabilized force

feedback, different methods have been implemented in surface-based

dental simulation systems.25 All of those methods tried to develop a

stable haptic interface system with higher frequencies. In this paper, a

2nd order bi-quadratic digital filter is adopted to reduce vibration and

obtain the smooth force feedback signal. The details are described as

follows:

(10)

where is the last position and is the updated position of

the penetrated vertex. Thus, the resultant force is the summation of all

forces:

(11)

(12)

(13)

where is the average normal vector, Nj is the vertex normal vector,

and m is the total number of vertices penetrated by the virtual cutter.

The mapping of phantom motion to the average normal direction of the

tooth surface is implemented by using Eq. 12 in which is the value

of projected onto . However, even after computing the resultant

force, low fidelity and vibration of the haptic stylus are issues that still

need to be resolved. Therefore, to resolve these issues, 2nd order bi-

quadratic digital filter is proposed. The discrete transfer function for the

z-transform is defined by:

Plocal M1– Pglobal=

P'local_x

P'local_y

P'local_z

r cosθ

r sinθ

Plocal_z

=0 d'Δ r< <

π θ π≤ ≤

θ 2atan Plocal_y

Plocal_x

,( )=

d'Δ Plocal_x

( )2 Plocal_y

( )2+=

Pglobal M P'local=

P'local_x

P'local_y

P'local_z

r hc⁄( )hp cosθ

r hc⁄( )hp sinθ

Plocal_z

= 0 d'Δ r hc⁄( )hp< <

F Ks xΔ⋅ Kd v⋅–=

xΔ v

xΔ Plast Pupdated–=

Plast Pupdated

Fr Ks( )i

xiΔ⋅i 0=

m

∑ Kd( )i

vp⋅–=

vp v n⋅( )n=

n Nj

j 0=

m

∑⎝ ⎠⎜ ⎟⎛ ⎞

Nj

j 0=

m

∑⎝ ⎠⎜ ⎟⎛ ⎞

⁄=

n

vp

v n

Fig. 6 Collision detection: (a) Cylindrical collision, (b) Cone collision


(14)

Hence the discrete transfer function is implemented with the

corresponding difference equation:

(15)

where the filter consists of n cascaded bi-quad sections, g[n] is the

overall gain of the filter, bk is the feedforward filter coefficients, aj is

the feedback filter coefficients, Fin[n] is the input force signal, and

Fout[n] is the output force signal. The parameters used with the filter

equation can be easily adjusted to the dynamic properties of the

phantom device, without changes in the algorithm. Finally, the filtered

force signal is sent to the haptic device controller for haptic rendering

at 1 kHz. The complete sculpting process with force exerted by

deformed vertices is shown in Fig. 7 in which Fig. 7(a) illustrates

separation state between the cutter and the tooth surface, Fig. 7(b)

depicts the intersection between the cutter and the tooth surface, Fig.

7(c) demonstrates the deformation of vertices from the original tooth

surface to the surface of the cutter after an intersection, Fig. 7(d) shows

the exerted force by deformed vertices, and Fig. 7(e) presents the final

tooth shape after cutting.

3.8 Bi-Tri Subdivision

During the real-time sculpting, it is observed that triangle

abnormalities occur along with deformations of tooth vertices, which

affect the tooth shape and vibration in the haptic device. In order to

handle this issue, a reconstruction method of triangles is used within

the local cutting area, in which a bi-tri subdivision of triangles is

utilized to correct abnormalities of triangles. According to the tri-

subdivision method, a midpoint of two longer edges is selected for the

division of the triangle from one to three triangles. At the same time,

the two adjacent triangles also need to be reconstructed for the stability

of the local mesh by a bi-subdivision method in which one triangle is

converted into two triangles. However, in a surface model, both

vertices and edges are shared by several adjacent triangles. Therefore,

six triangles are used to describe the twelve possible subdivision cases,

as illustrated in Fig. 8. In the first case of Fig. 7, suppose that 1- 2- 3

are the vertices and S1, S2, S3 are the edges of an irregular triangle.

First tri-subdivision is performed and two vertices 8 and 9 are inserted

into the edges S1 and S3, respectively. After that both adjacent triangles

1-3-4 and 1-7-2 are reconstructed through bi-subdivision. The vertices

8 and 9 are inserted when S1greater than S3 and S3 should be greater

than S2. Similarly, all the other cases are checked for the subdivision

of an abnormal triangle. In order to check which triangle lost its

normality behaviour within the local deformation area after the

collision and deformation of vertices, some basic triangle information

such as the area, incircle radius, and circumcircle radius of the only

local triangle is computed. The abnormality of the local triangle is

checked using predefined criteria, which is the ratio of incircle and

circumcircle of the local triangle. Regular and irregular behaviours of

the triangle can be seen in Fig. 9, where Rc is the circumcircle radius,

Rin is the incircle radius, and S1, S2, S3 are the lengths of edges of the

local triangle. In addition, if the case S1 = S2 = S3 occurs and if the

triangle does not meet the predefined criteria then 3-split subdivision is

performed on that triangle. In 3-split subdivision, one triangle is

converted into three triangles and only one centroid vertex is inserted.

After reconstructing the mesh in real-time, the model needs to be

updated due to changes in the position of vertices and the normal

information, therefore, the local updating algorithm is utilized because

of its efficiency and simplicity.

H z( ) g z( )b0 b1z

1–b2z

2–+ +

1 a1z1–

a2z2–

+ +---------------------------------------⋅=

Fout n[ ] g n[ ] bkFin n k–[ ]k 0=

2

∑ ajFout n j–[ ]j 1=

2

∑–⋅=

Fig. 7 Sculting process with force feedback: (a) No collision between

the cutter and the tooth surface, (b) Intersection between the cutter and

the tooth surface, (c) Deformation of verties from the original tooth

surface to the cutter’s surface after and intersection, (d) Force exerted

by deformed vertices, (e) The final tooth shape after cuttingFig. 8 Bi-tri subdivision: Twelve possible cases of subdivision

Fig. 9 Bi-tri Subdivision criteria: (a) Regular triangle, (b) Irregular triangle


3.9 Local Updating

After each sculpting stroke, the surface model must be updated in

real-time to enable correct collision detection in the next computation

cycle. In order to reduce the update time required for smooth haptic

sensation, local updating of the surface model is carried out because the

surface point data does not change very much during the sculpting

process. We only consider the deformed area for updating and

reconstructing.

3.10 Sound Rendering

In order to realize different sound effects during different virtual

dental operations, a simple approach is used in which various sound

effects were recorded from the real dental operations and stored in an

audio format. In order to play that audio file in the virtual environment,

an audio library is utilized. To realize the actual sound during material

removal operation, pitch, gain of the sound, and source and listener

positions are adjusted. The approximate result is achieved by using this

approach. However, the realistic physical contact sound can also be

improved in the future.

4. Experiments and Results

4.1 Hardware Platform

The adopted hardware setup of the proposed system is illustrated in

Fig. 10. The specifications include a 2.0 GHz Intel Core™2 CPU, 2

Giga Bytes of RAM, simulation rendering based on Nvidia GT220

GPU with 512 MB memory, a 22” monitor, two stereo speakers, and

windows based operating system Microsoft Windows Vista. The

Phantom Omni is used as a haptic device which offers three degree-of-

freedom output and six degree-of-freedom positional sensing

capabilities and was provided by SensAble Technologies.

4.2 System Architecture

The proposed virtual dental simulator has been implemented on a PC

using OpenGL graphics library to visualize 3-D models, OpenAL audio

library to realize drilling sound, and OpenHaptics toolkit to render the

force signal. The programming environment is Microsoft Visual Studio

C++ 2005. All predefined coefficients are selected according to the

maximum stiffness and force capability by Phantom OmniTM.

In this system, the simulation of material removal can be carried out

on any scanned surface model from a commercial 3-D dental scanner.

From our experiments, the computational calculation time was reduced

for real-time sculpting when the triangle subdivision algorithm was

used. All components, i.e. haptic rendering, visual rendering,

subdivision, collision detection, simulated sound, and model updating

were tested to ensure all functions work properly as per design

specifications. The testing procedure is summarized as follows:

1. Execute the system by importing tri-mesh file of a 3D human jaw

model in the virtual environment.

2. Perform the model reduction process and separate one tooth from

the model by creating two cutting planes.

3. Perform the mesh refinement process to make the model mesh

regular and initial communication between Phantom and software

was carried out.

4. Select the cutting tool with appropriate configurations. For the

verifying of frame rate, the dentist drives the cutting tool around the

tooth model freely along with the translation of the haptic stylus.

5. Perform the material removal operation by driving the haptic stylus

and increase the driving force until the material is removed from the

surface of the tooth. The depth of penetration of the cutter and the

force were then computed.

6. At the same time, the abnormality of triangles was checked by

finding neighbor triangles around the local triangle that need

enhancement and then the subdivision of the only local triangle was

performed.

7. After that, the model updating was carried out using local-based

updating method.

If there was something wrong in system stability or haptic

rendering, steps 3-7 were repeated.

4.3 Experiments and Results

First a model reduction algorithm was implemented by creating two

cutting planes on the jaw model. The original number of triangles was

5,992, the number of vertices was 3,103, and the percentage of

remaining data after data reduction was 4.99%. Next of mesh

refinement and real-time sculpting simulation using spherical and

cylindrical virtual tools were performed and the results are

demonstrated in Fig. 11. The three-dimensional material removal

results show that the visual and haptic update rates can be maintained

over standard criteria (30-60 Hz and 1 kHz). The sound of drilling in

material removal and in idle mode was also realized. Furthermore, two

modes were defined for the model manipulation. The first mode was

Fig. 10 Hardware setup of the system

Fig. 11 Material removed from teeth by implementing real-time drilling

and surface scrubbing using spherical and cylindrical virtual tools


the non-sculpted mode in which dentists can touch and feel the tissues

of the tooth, but cannot operate the sculpting process. The second was

a sculpted mode in which the dentist can cut, drill, or scrub the tooth.

In the implementation of three-dimensional sculpting, the results of

the virtual force feedback and the material removal operation were

evaluated in two experiments i.e., drilling and surface scrubbing. The

experiments can differ with respect to the surface conditions of the

static tooth model. For demonstrating the force signal, the magnitude of

the virtual force vector was recorded. Fig. 12 gives the interaction force

between the virtual cutter and the tooth surface in the real-time drilling

without force filtering. The force signal illustrates that the vibration of

the haptic stylus can be realized by the dentist’s hand and noise of

servo motors occur. In Fig. 13, the filtered smooth force signal can be

realized during the same drilling operation. However, the force fidelity

is decreased due to the force filtering; it can be optimized by adjusting

the value of the gain of the filter. Fig. 14 depicts the virtual force signal

of the real-time surface scrubbing before and after the force filtering.

The optimum coefficients that used for the force filtering were b0 = 1.0,

b1 = -0.1, b2 = -0.1, a1 = -0.2, a2 = -0.798, and g (gain) = 0.004. The

both successive drilling and surface scrubbing operations were

performed with force filter and are illustrated in Fig. 15. All

illustrations of the computed filtered force signal verify that the

realized force from deformed mesh is as expected for a surface-based

3D mesh model.

In order to test and verify the bi-tri subdivision algorithm, a

rectangle was used, as illustrated in Fig. 16(a). The collision between

the cutter and mesh vertices was essential for the material removal

process, due to this restriction Fig. 16(b) shows the tool interacting with

the rectangle at a corner vertex. The rectangle model was subdivided

from 12 to 212 triangles after sculpting operation. Fig. 17 depicts the

verified bi-tri subdivision algorithm during the real-time drilling of the

original tooth, and the new generated triangles were 404 after

subdivision. Furthermore, concerning the computation time of all

processes, the precision of the model reduction, mesh refinement, and

the exact collision between the virtual cutting tool and the surface of

the tooth was precisely investigated. Table 1 describes the computation

Fig. 12 Virtual force signal in the drilling operation without force filtering

Fig. 13 Virtual force signal in the drilling operation with force filtering

Fig. 14 Signal of the virtual force with and without force filtering

during the surface scrubbing operation

Fig. 15 Signal of the filtered force in drilling and surface scrubbing

operations

Fig. 16 Bi-tri subdivisions on a rectangle: (a) Beforeintersection, (b)

After intersection

Fig. 17 Bi-tri subdivisions on the original tooth mesh: (a) before real-

time drilling, (b) After real-time drilling


time of the model reduction and the mesh refinement algorithm. It is

important to mention here that the mesh refinement process was

implemented before the real-time haptic rendering. Therefore, there is

not any influence of the refinement process on the real-time material

removal rendering; thus, increased the frame rate in the real-time haptic

rendering and reduced the haptic stylus vibration. The table data also

shows the comparison between loop and subdivisions, and it is

concluded that the subdivision creates less number of triangles but

takes more time than the loop subdivision. Because the mesh

refinement process occurs before the real-time haptic rendering and

time was not the issue at that stage, therefore, it is acceptable for

dentists and their candidates. In addition, the Table 2 gives the

computation time of real-time collision between the surface of the tooth

and the virtual tool that shows considering the more number of

triangles in the real-time haptic rendering results more computation

time and less frame rate. Therefore, it was required to reduce the model

data for fast haptic rendering. Finally, the results from the experiment

verify that this system can generate stable three-dimensional sculpting

simulation from a human tooth model with realistic auditory, visual,

and haptic sensations.

5. Discussion

Haptic devices allow dentists to perform different operations like

pulling, pushing, and cutting of soft or hard tissue with realistic force

feedback. During tooth cutting operation, too much applied force will

increase the rate of heat generation and thus damage the tooth tissues,

while too little force may prolong the painful treatment procedure for

the patient. Therefore, haptic sensation is crucial for the dentists to

operate successfully.

Trials were performed with 20 users performing 15 sessions. Each

user was given instructions for performing the drilling and surface

scrubbing operations. This practice was repeated once with 20 users.

From the results, it was also observed that the speed of the virtual

cutting tool should be realistic because fast phantom speed could create

instabilities in the force computation and model mesh. As seen in Fig.

14, the vibration of the haptic stylus occurs. This could be due to the

vibrations in the wrist and the hand instability. Therefore, we tried to

solve this issue by introducing a force filter and two modes that control

the tool’s speed during material removal simulation: the low speed

mode is activated when the tool moves fast and otherwise the normal

speed mode is active.

Since, there is no internal part of the surface-based jaw model;

therefore, the system is restricted to perform sculpting simulation on

the surface. We only consider the material’s surface properties for

sensing the force feedback. In order to consider deep drilling operation

or if the user continues to drill, the system may become unstable due

to overlapping of triangles. To solve this problem, bi-tri subdivision

with predefined criteria for abnormal triangles can have reasonable

sculpting and visual results. However, future work should address this

problem for stabilizing the system.

Fidelity is defined as the force feeling during virtual cutting

operation is similar to that in real dental operation, and stability refers

to the stable running of the haptic device with high update rates. It is

observed that there is always a large difference for the fidelity and

stability of haptics simulation compared with real dental operation and

that unstable haptic sensation in dental training can mislead the dentists

in real operation. Therefore, fidelity and stability of haptics simulation

are very important for dental sculpting operation. In this system, in

order to achieve high update rate, model reduction, local based data

updating, and simple force model was adopted. To achieve suitable

fidelity, the digital filter was proposed with the gain factor. User can

manually adjust the gain of the filter for stable and smooth force

feedback.

6. Conclusion

In this paper, a surface-based virtual dental sculpting system has

presented using mixed realities in which dentists can perform different

dental procedures with smooth tactile feeling, auditory and visual

realization. Different dental treatment procedures in this proposed

simulator could help dentists to perform safer operations before

entering to the real operation task. A model reduction algorithm is

proposed in which dentists can separate the desired area of the tooth

from the human jaw model for fast haptic rendering. Different mesh

refinement techniques are utilized to make the model mesh regular. A

fast vertex deformation method is used to simulate the removal of tooth

tissues. To prevention abnormalities of triangles during deformation of

vertices, detailed twelve possible cases of the bi-tri subdivision are

proposed. Enhanced model updating is used to achieve the display

update rate 30-60 Hz using a local-based updating method.

Experiments are carried out using a Phantom OmniTM haptic device;

3

3

Table 1 Computation time of model reduction and mesh refinement

algorithms before real-time haptic rendering

Model

TrianglesExp.

Triangles

after model

reduction

Model

reduction

time (sec)

Triangles

after mesh

refinement

(one round)

Mesh

refinement

time (sec)

119995

11645

half tooth0.075

6748 – L

5209 – S

0.01 – L

0.08 – S

24419

one tooth0.090

17676 – L

13451 – S

0.11 – L

0.16 – S

39698

two teeth0.110

38792 – L

29377 – S

0.92 – L

1.10 – S

L = Loop subdivision, S = 3 subdivision

Table 2 Computation time of real-time collision after 3 subdivision

Exp. Triangles

Collision detection

+ force computation

time (sec)

Results

1 5209

0.009 - sphere

0.010 - cylinder

0.012 - cone

i) Higher frame rate

ii) Smooth haptic rendering

2 13451

0.011 - sphere

0.019 - cylinder

0.022 - cone

i) Higher frame rate

ii) Smooth haptic rendering

3 29377

0.018 - sphere

0.021 - cylinder

0.031 - cone

i) Less frame rate

ii) Haptic vibration occurs


these experiments involve the three-dimensional cutting of a tooth

model using a 2nd order bi-quadratic force filter. The experiment

operations verify that the force stability can be easily maintained under

the specified operation criteria.

In the future, we plan to enhance the simulator by improving the

stability, fidelity, realistic drilling sound further. We also have plan to

add augmented reality (AR), cleaning, bleeding, evaluation of student

performance, and recoding the training procedure performed by dental

students.

ACKNOWLEDGEMENT

This work was supported in part by the Korea Ministry of

Knowledge Economy, under Grant of the Strategic Technology

Development Project on Biomedical Supplier (Development of the

Digital Fusion Artificial Tooth Treatment Supporting System).

REFERENCES

1. Bruyns, C. and Senger, S., “Interactive Cutting of 3-D Surface

Meshes,” Computer and Graphics, Vol. 25, No. 4, pp. 635-642,

2001.

2. Zhang, H. and Payandeh, D. S. J., “Simulation of Progressive

Cutting on Surface Mesh model,” DRAFT 6-08, pp. 1-8, 2002.

3. Brown, J., Sorkin, S., Latombe, J., and Montgomery, K.,

“Algorithmic Tools for Real-Time Microsurgery Simulation,”

Medical Image Analysis, Vol. 6, No. 3, pp. 289-300, 2002.

4. Choi, K. S., “Interactive cutting of deformable objects using force

propagation approach and digital design analogy,” Computer and

Graphics, Vol. 30, pp. 233-243, 2006.

5. Dachille, F., Qin, H., and Kaufman, A. E., “A novel haptic-based

interface and sculpting system for physical based geometric design,”

Compter-Aided Design, Vol. 33, No. 5, pp. 403-420, 2001.

6. Chang, Y. H., Chen, Y. T., Chang, C. W., and Lin, C. L.,

“Development scheme of haptic-based system for interactive

deformable simulation,” Computer-Aided Design, Vol. 42, pp. 414-

424, 2010.

7. Basdogan, C., De, S., Kim, J., Muniyandi, M., Kim, H., and

Srinivasan, M. A., “Haptics in minimally invasive surgical

simulation and training,” IEEE Computer Graphics and

Applications, Vol. 24, No. 2, pp. 56-64, 2004.

8. Rodrigues, M. A. F., Silva, W. B., Neto, M. E. B., Gillies, D. F., and

Ribeiro, I. M. M. P., “An interactive simulation system for training

and treatment planning in orthodontics,” Computer and Graphics,

Vol. 31, pp. 688-697, 2007.

9. Thomas, G., Johnson, L., and Stanford, D., “The Design and Testing

of a Force Feedback Dental Simulator,” Computer Methods and

Programs in Biomedicine, Vol. 11, No. 64, pp. 53-64, 2001.

10. Luciano, C., Banerjee, P., and DeFanti, T., “Haptics-based virtual

reality periodontal training simulator,” Virtual Reality, Vol. 13, No.

2, pp. 69-85, 2009.

11. Rhienmora, P., Gajananan, K., Haddawy, P., Dailey, M. N., and

Suebnukarn, S., “Augmented Reality Haptics System for Dental

Surgical Skills Training,” VRST 2010, pp. 97-98, 2010.

12. Noborio, H., Sasaki, D., Kawamoto, Y., Tatsumi, T., and Sohmura,

T., “Mixed reality software for dental simulation system,” IEEE

International Workshop on Haptic Audio Visual Environments and

their applications, pp. 19-24, 2008.

13. Bogdan, C. M., “Domain ontology of the VirDenT system,” J.

Comput. Commun. Control, Vol. 6, pp. 45-52, 2011.

14. Wang, D., Zhang, Y., Hou, J., Wang, Y., Lv, P., Chen, Y., and Zhao,

H., “iDental: a haptic-based dental simulator and its preliminary user

evaluation,” IEEE Trans. Haptics, Vol. 5, No. 4, pp. 332-343, 2011.

15. Laehyun, K., Gaurav, S. S., and Mathieu, D., “A Haptic Rendering

Technique Based on Hybrid Surface Representation,” IEEE

Computer Graphics and Applications, Vol. 24, No. 2, pp. 66-75,

2004.

16. Kim, L. and Park, S., “An efcient teeth modeling for dental training

system,” Int. J. CAD/CAM, Vol. 8, No. 1, pp. 41-44, 2008.

17. Kimin, K. and Park, J., “Virtual bone drilling for dental implant

surgery training,” VRST 2009, 2009.

18. Novint Technologies, “Virtual reality dental training system (vrdts),”

http://www.novint.com/index.php/apg/medicaldental, [Accessed:

Sept. 2012]

19. Marras, I., Papaleontiou, L., Nikolaidis, N., Lyroudia, K., and Pitas,

I., “Virtual Dental Patient: a System for Virtual Teeth Drilling,”

IEEE International Conference on Multimedia and Expo., 2006.

20. Noborio, H., Sasaki, D., Kawamoto, Y., Tatsumi, T., and Sohmura,

T., “Construction of Dental Simulation System with Mixed Visual,

Tactile, and Sound Realities,” 18th International Conference on

Artificial Reality and Telexistence, 2008.

21. Chang, H. C., Jin, Y. L., Yong, K. L., and Mun, T. C., “Determining

the Passive Region of the Multirate Wave Transform on the Practical

Implementation,” Int. J. Precis. Eng. Manuf., Vol. 12, No. 6, pp.

975-981, 2011.

22. Park, J., Kim, K., and Hong, D., “Haptic-based resistance training

machine and its application to biceps exercises,” Int. J. Precis. Eng.

Manuf., Vol. 12, No. 1, pp. 21-30, 2011.

23. Ullah, F. and Park, K., “Virtual Dental Sculpting Simulation using a

Surface-based Tooth Model and a Haptic Device,” Korean Society

of CAD/CAM Engineers, pp. 838-851, 2011.

24. Ullah, F. and Park, K., “Visual, Haptic, and Auditory Realities based

Dental Training Simulator,” 2012 International Conference on

Information Science and Applications, pp. 106-111, 2012.

25. Ullah, F. and Park, K., “Surface-Based Virtual Dental Surgical


Simulator using Haptic Display,” Computer-Aided Design &

Applications, Vol. 8, No. 6, pp. 841-848, 2011.

26. Ullah, F. and Park, K., “Surface-based Virtual Dental Sculpting

Simulation with Multimodal Feedback,” Asian Conference on

Design and Digital Engineering, pp. 548-555, 2011.

27. Ullah, F. and Park, K., “Virtual Dental Treatment Training System

using a Haptic Device,” Korean Society of CAD/CAM Engineers,

pp. 115-121, 2009.

28. Xia, P., Lopes, A., and Restivo, M., “Virtual reality and haptics for

dental surgery: a personal review,” The Visual Computer, pp. 1-15,

2012.

29. Ullah, F., Lee, G. S., and Park, K., “Piezoelectric Transducer based

3D Intraoral Scanner,” 2012 International Conference on Information

Science and Applications, pp. 118-123, 2012.

30. Ullah, F., Lee, G. S., and Park, K., “Development of a Real-time 3D

Intraoral Scanner based on Fringe-Projection Technique,”

Transactions of the Society of CAD/CAM Engineers, Vol. 17, No. 3,

pp. 156-163, 2012.

31. Kobbelt, L., “3 Subdivision,” Proc. of SIGGRAPH, pp. 103-112,

2000.

32. Loop, C. T., “Smooth subdivision surfaces based on triangles,”

Master’s Thesis, Department of Mathematics, University of Utah,

1987.

33. Bischoff, S. and Kobbelt, L., “Teaching meshes, subdivision and

multiresolution techniques,” Computer-Aided Design, Vol. 36, No.

14, pp. 1483-1500, 2004.

34. Foskey, M., Otaduy, M. A., and Lin, M. C., “ArtNova: Touch-

enabled 3D model design,” IEEE Virtual Reality Proceedings, pp.

119-126, 2002.

35. McDonnell, K. T., Qin, H., and Wlodarczyk, R. A., “Virtual clay: a

real-time sculpting system with haptic toolkits,” ACM Symposium

on Interactive 3D Techniques, pp. 179-190, 2001.

36. Tanaka, A., Hirota, K., and Kaneko, T., “Virtual cutting with force

feedback,” Proc. of Virtual Reality Annual International

Symposium, Vol. 199, No. 8, pp. 71-77, 1998.

37. Tse, B., Harwin, W., Barrow, A., Quinn, B., San Diego, J. P., and

Cox, M., “Design and development of a haptic dental training

system - hapTEL,” Euro Haptics Conference, pp. 101-108, 2010.

Date post:	13-Dec-2016
Category:	Documents
Upload:	kang-park
View:	216 times
Download:	3 times

Development of a surface-based virtual dental sculpting simulator with multimodal feedback

Documents