Positioning of Drug Carriers usingPermanent Magnet-Based Robotic System

Master Thesisof

cand M Sc Abdelrahman Hassan ElfarApplication number 22-0694

Field of study Mechatronics EngineeringDuration of work 6 MonthsSubmitted on 15032017Number S 1712Supervisor Dr Islam S M KhalilExternal supervisor Dr Andre Duffe

I declare that this work made in the ordinary course of self-care at the Institute usedto have and no other than the specified sources

(Abdelrahman Hassan Elfar)

Cairo 15032017

iv

Contents

Acknowledgement 1

Abstract 3

1 Introduction 511 Motivation 512 Electromagnetic Systems and Magnetic Control 513 Thesis Structure 9

2 Experiments preparation 1121 Modeling of a KUKA IIWA Serial-Link Manipulator 11

211 Robots in Minimally Invasive Interventions 11212 KUKA iiwa r800 12213 Interface of the KUKA robot 12

2131 Communication protocols for connecting the robot serverwith a c application 14

214 Additional features added to the robot arm to allow a free mo-tion in the null frame and motion to a target position 15

21401 Gravity compensation feature of allowing robotarm motion in the null space 16

21402 Motion feature for driving the robot arm toan aimed position 16

22 Camera calibration to extract the intrinsic parameters 19221 Camera modelling 20

23 Feature Tracking of Microparticles 2724 Kinematic Modeling of the Robotic Arm and Microparticle 2925 Control of the Magnetic-Based Robotic System 34

v

Contents

3 Experiments 3731 Position control of paramagnetic microparticles in xz-plane 38

311 Upward movement of the cluster 39312 Downward movement of the cluster 40

32 Position control of paramagnetic microparticles 40321 Motion control trials 40

33 Control of paramagnetic microparticle along a trajectory 42

4 Conclusions and Future work 4941 Conclusions 5042 Future work 50

Bibliography 51

List of Figures 61

List of Tables 67

Appendix 691 Communication protocol User Datagram Protocol (UDP) 69

1001 Sending bytes message from robot server to C 691002 Receiving byte message from C application

in the robot server 712 Creating a UDP communication in C application 723 Recieving a byte message using udp in java and C 734 Sending a byte message using udp in java and C 735 Image processing application example 73

vi

Acknowledgement

I would like to express my gratitude to my supervisor Dr Islam khalil for the usefulcomments remarks and engagement through the learning process of this master thesisFurthermore I would like to thank Dr Andre duffe for introducing me to the topic aswell for the support on the way

I would like to thank my parents and friends who have supported me throughout entireprocess both by keeping me harmonious and helping me putting pieces together Iwill be grateful forever for your love

1

Abstract

Magnetic control of drug carriers using systems with open-configurations is essentialto enable scaling to the size of in vivo applications In this study we demonstratemotion control of paramagnetic microparticles in a low Reynolds number fluid using apermanent magnet-based robotic system with an open-configuration The microparti-cles are controlled in three-dimensional (3D) space using a cylindrical NdFeB magnetthat is fixed to the end-effector of a robotic arm (Kuka iiwa r800 light weight robotwith 7 degrees of freedom and built in safety for better human interaction environ-ment) We develop a kinematic map between the position of the microparticles andthe configuration of the robotic arm and use this map as a basis of a closed-loop con-trol system based on the position of the microparticles Our closed-loop experimentalresults show the ability of the robot configuration to control the exerted magnetic fieldgradient on the dipole moment of the microparticles and achieve positioning in 3Dspace with maximum error of 300 microm and 600 microm in the steady-state during setpointand trajectory tracking respectively

3

1

Introduction

11 Motivation

Cancer is considered as the world second leading cause of death after heart disease [1]Survival rates are increasing owing to the improved medical treatment However thecurrent methods of treatment have negative side-effects on the body such as painloss of balance malnutrition and blood disorder Therefore much effort has beenexpanded to achieve targeted therapy that can treat cancer more safely than conven-tional methods [2] This work investigates the utlilzation of magnetic drug carriers(paramagnetic microparticles) that can reach the targeted position via a magnetic-based robotic system This study is necessary to translate the microparticles intoin vivo application

12 Electromagnetic Systems and Magnetic Control

Magnetically-driven and self-driven medical microrobots have the potential to deliverdrugs to deep seated regions in the human body Paramagnetic microparticles andhelical robots are examples of magnetically-driven microrobots while magnetotacticbacteria is a type of self-driven microrobots that have the potential to controllably

5

1 Introduction

Figure 11 The OctoMag prototype is designed and constructed at ETH Zurich Thesystem contains eight electromagnets in closed configuration [6]

achieve targeted drug delivery [3] A study was made for magnetic nanoparticles withsurface bound drug molecules type of microrobots were injected into the vascular sys-tem upstream from malignant tissue and were captured at the tumor using a localapplied magnetic field [4] Iron-oxide nanoparticles have been used in vitro by achiev-ing selective penetration of MCF-7 breast cancer cell without causing any damage tothe memberane and without any effect on the cell morphology [5] These differenttypes of microrobots are controlled wirelessly using controlled magnetic fields whichcan be generated using several methods The first method uses electromagnets thatare simultaneously position and current controlled The second method uses position-controlled permanent magnets Closed-configurations systems are generally thoughtto be essential for diverse biomedical applications during in vitro experimentation [7][8] The OctoMag shown in Fig 11 is a type of closed-loop control magnetic systemwith computer vision implemented for accurate positioning [2] [6] The OctoMag isdesigned for controlling the intraocular microrobots in minimally invasive retinal ther-apy and diagnosis It is also has been used as a wireless micromanipulation systemunder a light microscope Due to several technological barriers these systems cannotbe scaled up to the size of invivo applications [9] as it is generally not viable to achieverelatively high gradients at a distance They are placed in a way to form a closed-

6

12 Electromagnetic Systems and Magnetic Control

Figure 12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltechnolo-gie GmbH Rostock-Warnemuende Germany) in three-dimensional spaceThe magnetic system consists of 8 iron-core electromagnets that surroundsa water reservoir Magnetic-based robotic system give more freedom inmoving through trajectories compared to this system However this sys-tem showed greater positioning accuracy compared to the magnet-basedone [36]

configuration where the workspace containing microrobot which can be in the formof paramagnetic microparticles and nanoparticles The advantage of these particlesover the other types of microrobots is that they are biodegradable which give themthe opportunity to be used in most clinical applications However the magnetic forcesextended on these particles are small due to the dependency of the magnetic force onthe size and geometry of the object itself This problem can be solved by increasingthe directional derivatives in the applied field found between these coils (usually 4 or 8electromagentic coils) as shown Fig 12 The limited projection distance of the mag-netic field gradient constrains the workspace of these systems to tens of millimeters atmost

It is also difficult to integrate a clinical imaging modality to magnetic systems with

7

1 Introduction

closed configuration [10] [13] for instance These challenges have been partially over-come by Mahoney et al [9] [11] A combination of magnetic field-driven helicalmicrorobots [12] [15] and a robotic arm with a fixed permanent magnet to its endef-fector has been proposed and used to achieve locomotion throughout relatively largeworkspace in 3D space The rotating dipole field enables the microrobot to achievehelical propulsion while the robotic arm allows the rotating permanent magnet togenerate the rotating fields at any point within its workspace Recently Nelson etal [16] have utilized the non-uniformity of the rotating field in generating two inde-pendent rotating magnetic fields using a single magnet dipole An open configurationof two synchronized rotating dipole fields has been also introduced by Alshafeei etal to control the motion of helical microrobot inside a catheter segment [17] and in3D space [18] The open configuration of this magnetic-based robotic system enablesscaling to the size of in vivo applications and its ability to remove blood clots hasbeen demonstrated in vitro [19] An electromagnetic coil and a permanent magnethave been integrated to the end-effector of a robotic arm with four degrees-of-freedom(DOFs) in [20] and a comparative study has shown the ability of the coil to achievehigher positioning accuracy of paramagnetic microparticles compared to the perma-nent magnet Although electromagnetic coils provide greater positioning accuracyowing to the ability to control the magnitude of the field gradient via the currentinput it is desirable to benefit from the higher magnetic field and field gradient thatcan be generated using permanent magnets A theoretical analysis and experimen-tal tele-manipulation of trapped paramagnetic micro-particles inside gas bubbles wasintroduced by our research group Khalil et al by analyzing the interaction forces ofa trapped micro-particle in a gas bubble at the water-gas interface and estimatingthe interaction forces between the micro-particles and the bubble using a calibratedforce observer [22] Also another study was conducted for expermintly investigationthe non-contact manipulation of the microbeads using a cluster of microparticles Thenon-contact micromanipulation achieves successful releases by moving the microbeadstowards the reference positions [23]In this work we design a closed-loop control system to position clusters of param-agnetic microparticles in low-Reynolds number fluids The control is achieved in 3Dspace using a robotic arm with seven DOFs and a permanent magnet First we derivea mapping between the position of the cluster of paramagnetic microparticles and theconfiguration of the robotic arm This mapping is used as basis of a closed-loop controlsystem to position the cluster in 3D space

8

13 Thesis Structure

13 Thesis Structure

This thesis presents a control frame work to achieve positioning of drug carriers (para-magnetic magneticparticles) in 3D space using a magent-based robot in a closed-loopconfiguration The thesis is structured as follows

1 Chapter 2 describes the motion control and modeling of the paramagnetic mi-croparticles in 3D space

bull Section 21 presents the interface of the robotic arm (KUKA iiwa) thecommunication protocol used and the two sides of the interface in java andC

bull Section 22 explains the camera calibration to extract the internsic param-eters of the two cameras used in the experiments

bull Section 23 provides the image processing used to track the particle in real-time

bull Section 24 provides the kinematic model of the robotic arm and the mi-croparticles and the closed-loop control system design

2 Chapter 3 presents the experiments part showing the different positioning ofthe particles in the 3D space

bull Section 31 provides the control of a cluster of particles in xy-plane

bull Section 32 provides the control of a cluster of paricles in yz-plane

bull Section 33 provides the motion of the particles while following a line tra-jectory

bull Section 34 provides the motion of the particles while following a sinwave

3 Chapter 4 concludes and provides directions for future work

bull Section 41 provides the conclusions of this work

bull Section 42 discusses suggestions pertaining to our future work

9

1 Introduction

4 Chapter 5 presnsents the appendix and references used for this thesis

10

2

Experiments preparation

21 Modeling of a KUKA IIWA Serial-Link Manipulator

KUKA iiwa R800 (7 DOFs robot KUKA GmbH Germany) was used in this workand an interface was implemented to control the robot

211 Robots in Minimally Invasive Interventions

The range of applications for medical robots has increased in recent years Hospitalsreport hundreds and even thousands of procedures performed with the da Vinci robotshown in Fig21 which now has the delicate ability to precisely peel a grape It isused in all kinds of surgeries from heart-valve repair to tumor removal and acrossvarious surgical disciplines The ROBODOC shown in Fig22 now combines withORTHODOC to preplan surgical procedures and carry them out with relatively highaccuracy ROBODOC also assists the physicians with knee replacements Roboticprosthetics now include sensors that are attached to the muscles and nerves allowingpatients to experience a sense of touch and to control movements with thought Pros-thetic hands allow fine motor skills such as writing typing or playing piano Robotsmight have feelings after all with the help of a robotic exoskeleton stroke victimscan regain lost arm movement quicker than with traditional physical therapy Able to

11

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

I declare that this work made in the ordinary course of self-care at the Institute usedto have and no other than the specified sources

(Abdelrahman Hassan Elfar)

Cairo 15032017

iv

Contents

Acknowledgement 1

Abstract 3

1 Introduction 511 Motivation 512 Electromagnetic Systems and Magnetic Control 513 Thesis Structure 9

2 Experiments preparation 1121 Modeling of a KUKA IIWA Serial-Link Manipulator 11

211 Robots in Minimally Invasive Interventions 11212 KUKA iiwa r800 12213 Interface of the KUKA robot 12

2131 Communication protocols for connecting the robot serverwith a c application 14

214 Additional features added to the robot arm to allow a free mo-tion in the null frame and motion to a target position 15

21401 Gravity compensation feature of allowing robotarm motion in the null space 16

21402 Motion feature for driving the robot arm toan aimed position 16

22 Camera calibration to extract the intrinsic parameters 19221 Camera modelling 20

23 Feature Tracking of Microparticles 2724 Kinematic Modeling of the Robotic Arm and Microparticle 2925 Control of the Magnetic-Based Robotic System 34

v

Contents

3 Experiments 3731 Position control of paramagnetic microparticles in xz-plane 38

311 Upward movement of the cluster 39312 Downward movement of the cluster 40

32 Position control of paramagnetic microparticles 40321 Motion control trials 40

33 Control of paramagnetic microparticle along a trajectory 42

4 Conclusions and Future work 4941 Conclusions 5042 Future work 50

Bibliography 51

List of Figures 61

List of Tables 67

Appendix 691 Communication protocol User Datagram Protocol (UDP) 69

1001 Sending bytes message from robot server to C 691002 Receiving byte message from C application

in the robot server 712 Creating a UDP communication in C application 723 Recieving a byte message using udp in java and C 734 Sending a byte message using udp in java and C 735 Image processing application example 73

vi

Acknowledgement

I would like to express my gratitude to my supervisor Dr Islam khalil for the usefulcomments remarks and engagement through the learning process of this master thesisFurthermore I would like to thank Dr Andre duffe for introducing me to the topic aswell for the support on the way

I would like to thank my parents and friends who have supported me throughout entireprocess both by keeping me harmonious and helping me putting pieces together Iwill be grateful forever for your love

1

Abstract

Magnetic control of drug carriers using systems with open-configurations is essentialto enable scaling to the size of in vivo applications In this study we demonstratemotion control of paramagnetic microparticles in a low Reynolds number fluid using apermanent magnet-based robotic system with an open-configuration The microparti-cles are controlled in three-dimensional (3D) space using a cylindrical NdFeB magnetthat is fixed to the end-effector of a robotic arm (Kuka iiwa r800 light weight robotwith 7 degrees of freedom and built in safety for better human interaction environ-ment) We develop a kinematic map between the position of the microparticles andthe configuration of the robotic arm and use this map as a basis of a closed-loop con-trol system based on the position of the microparticles Our closed-loop experimentalresults show the ability of the robot configuration to control the exerted magnetic fieldgradient on the dipole moment of the microparticles and achieve positioning in 3Dspace with maximum error of 300 microm and 600 microm in the steady-state during setpointand trajectory tracking respectively

3

1

Introduction

11 Motivation

Cancer is considered as the world second leading cause of death after heart disease [1]Survival rates are increasing owing to the improved medical treatment However thecurrent methods of treatment have negative side-effects on the body such as painloss of balance malnutrition and blood disorder Therefore much effort has beenexpanded to achieve targeted therapy that can treat cancer more safely than conven-tional methods [2] This work investigates the utlilzation of magnetic drug carriers(paramagnetic microparticles) that can reach the targeted position via a magnetic-based robotic system This study is necessary to translate the microparticles intoin vivo application

12 Electromagnetic Systems and Magnetic Control

Magnetically-driven and self-driven medical microrobots have the potential to deliverdrugs to deep seated regions in the human body Paramagnetic microparticles andhelical robots are examples of magnetically-driven microrobots while magnetotacticbacteria is a type of self-driven microrobots that have the potential to controllably

5

1 Introduction

Figure 11 The OctoMag prototype is designed and constructed at ETH Zurich Thesystem contains eight electromagnets in closed configuration [6]

achieve targeted drug delivery [3] A study was made for magnetic nanoparticles withsurface bound drug molecules type of microrobots were injected into the vascular sys-tem upstream from malignant tissue and were captured at the tumor using a localapplied magnetic field [4] Iron-oxide nanoparticles have been used in vitro by achiev-ing selective penetration of MCF-7 breast cancer cell without causing any damage tothe memberane and without any effect on the cell morphology [5] These differenttypes of microrobots are controlled wirelessly using controlled magnetic fields whichcan be generated using several methods The first method uses electromagnets thatare simultaneously position and current controlled The second method uses position-controlled permanent magnets Closed-configurations systems are generally thoughtto be essential for diverse biomedical applications during in vitro experimentation [7][8] The OctoMag shown in Fig 11 is a type of closed-loop control magnetic systemwith computer vision implemented for accurate positioning [2] [6] The OctoMag isdesigned for controlling the intraocular microrobots in minimally invasive retinal ther-apy and diagnosis It is also has been used as a wireless micromanipulation systemunder a light microscope Due to several technological barriers these systems cannotbe scaled up to the size of invivo applications [9] as it is generally not viable to achieverelatively high gradients at a distance They are placed in a way to form a closed-

6

12 Electromagnetic Systems and Magnetic Control

Figure 12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltechnolo-gie GmbH Rostock-Warnemuende Germany) in three-dimensional spaceThe magnetic system consists of 8 iron-core electromagnets that surroundsa water reservoir Magnetic-based robotic system give more freedom inmoving through trajectories compared to this system However this sys-tem showed greater positioning accuracy compared to the magnet-basedone [36]

configuration where the workspace containing microrobot which can be in the formof paramagnetic microparticles and nanoparticles The advantage of these particlesover the other types of microrobots is that they are biodegradable which give themthe opportunity to be used in most clinical applications However the magnetic forcesextended on these particles are small due to the dependency of the magnetic force onthe size and geometry of the object itself This problem can be solved by increasingthe directional derivatives in the applied field found between these coils (usually 4 or 8electromagentic coils) as shown Fig 12 The limited projection distance of the mag-netic field gradient constrains the workspace of these systems to tens of millimeters atmost

It is also difficult to integrate a clinical imaging modality to magnetic systems with

7

1 Introduction

closed configuration [10] [13] for instance These challenges have been partially over-come by Mahoney et al [9] [11] A combination of magnetic field-driven helicalmicrorobots [12] [15] and a robotic arm with a fixed permanent magnet to its endef-fector has been proposed and used to achieve locomotion throughout relatively largeworkspace in 3D space The rotating dipole field enables the microrobot to achievehelical propulsion while the robotic arm allows the rotating permanent magnet togenerate the rotating fields at any point within its workspace Recently Nelson etal [16] have utilized the non-uniformity of the rotating field in generating two inde-pendent rotating magnetic fields using a single magnet dipole An open configurationof two synchronized rotating dipole fields has been also introduced by Alshafeei etal to control the motion of helical microrobot inside a catheter segment [17] and in3D space [18] The open configuration of this magnetic-based robotic system enablesscaling to the size of in vivo applications and its ability to remove blood clots hasbeen demonstrated in vitro [19] An electromagnetic coil and a permanent magnethave been integrated to the end-effector of a robotic arm with four degrees-of-freedom(DOFs) in [20] and a comparative study has shown the ability of the coil to achievehigher positioning accuracy of paramagnetic microparticles compared to the perma-nent magnet Although electromagnetic coils provide greater positioning accuracyowing to the ability to control the magnitude of the field gradient via the currentinput it is desirable to benefit from the higher magnetic field and field gradient thatcan be generated using permanent magnets A theoretical analysis and experimen-tal tele-manipulation of trapped paramagnetic micro-particles inside gas bubbles wasintroduced by our research group Khalil et al by analyzing the interaction forces ofa trapped micro-particle in a gas bubble at the water-gas interface and estimatingthe interaction forces between the micro-particles and the bubble using a calibratedforce observer [22] Also another study was conducted for expermintly investigationthe non-contact manipulation of the microbeads using a cluster of microparticles Thenon-contact micromanipulation achieves successful releases by moving the microbeadstowards the reference positions [23]In this work we design a closed-loop control system to position clusters of param-agnetic microparticles in low-Reynolds number fluids The control is achieved in 3Dspace using a robotic arm with seven DOFs and a permanent magnet First we derivea mapping between the position of the cluster of paramagnetic microparticles and theconfiguration of the robotic arm This mapping is used as basis of a closed-loop controlsystem to position the cluster in 3D space

8

13 Thesis Structure

13 Thesis Structure

This thesis presents a control frame work to achieve positioning of drug carriers (para-magnetic magneticparticles) in 3D space using a magent-based robot in a closed-loopconfiguration The thesis is structured as follows

1 Chapter 2 describes the motion control and modeling of the paramagnetic mi-croparticles in 3D space

bull Section 21 presents the interface of the robotic arm (KUKA iiwa) thecommunication protocol used and the two sides of the interface in java andC

bull Section 22 explains the camera calibration to extract the internsic param-eters of the two cameras used in the experiments

bull Section 23 provides the image processing used to track the particle in real-time

bull Section 24 provides the kinematic model of the robotic arm and the mi-croparticles and the closed-loop control system design

2 Chapter 3 presents the experiments part showing the different positioning ofthe particles in the 3D space

bull Section 31 provides the control of a cluster of particles in xy-plane

bull Section 32 provides the control of a cluster of paricles in yz-plane

bull Section 33 provides the motion of the particles while following a line tra-jectory

bull Section 34 provides the motion of the particles while following a sinwave

3 Chapter 4 concludes and provides directions for future work

bull Section 41 provides the conclusions of this work

bull Section 42 discusses suggestions pertaining to our future work

9

1 Introduction

4 Chapter 5 presnsents the appendix and references used for this thesis

10

2

Experiments preparation

21 Modeling of a KUKA IIWA Serial-Link Manipulator

KUKA iiwa R800 (7 DOFs robot KUKA GmbH Germany) was used in this workand an interface was implemented to control the robot

211 Robots in Minimally Invasive Interventions

The range of applications for medical robots has increased in recent years Hospitalsreport hundreds and even thousands of procedures performed with the da Vinci robotshown in Fig21 which now has the delicate ability to precisely peel a grape It isused in all kinds of surgeries from heart-valve repair to tumor removal and acrossvarious surgical disciplines The ROBODOC shown in Fig22 now combines withORTHODOC to preplan surgical procedures and carry them out with relatively highaccuracy ROBODOC also assists the physicians with knee replacements Roboticprosthetics now include sensors that are attached to the muscles and nerves allowingpatients to experience a sense of touch and to control movements with thought Pros-thetic hands allow fine motor skills such as writing typing or playing piano Robotsmight have feelings after all with the help of a robotic exoskeleton stroke victimscan regain lost arm movement quicker than with traditional physical therapy Able to

11

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

Cairo 15032017

iv

Contents

Acknowledgement 1

Abstract 3

1 Introduction 511 Motivation 512 Electromagnetic Systems and Magnetic Control 513 Thesis Structure 9

2 Experiments preparation 1121 Modeling of a KUKA IIWA Serial-Link Manipulator 11

211 Robots in Minimally Invasive Interventions 11212 KUKA iiwa r800 12213 Interface of the KUKA robot 12

2131 Communication protocols for connecting the robot serverwith a c application 14

214 Additional features added to the robot arm to allow a free mo-tion in the null frame and motion to a target position 15

21401 Gravity compensation feature of allowing robotarm motion in the null space 16

21402 Motion feature for driving the robot arm toan aimed position 16

22 Camera calibration to extract the intrinsic parameters 19221 Camera modelling 20

23 Feature Tracking of Microparticles 2724 Kinematic Modeling of the Robotic Arm and Microparticle 2925 Control of the Magnetic-Based Robotic System 34

v

Contents

3 Experiments 3731 Position control of paramagnetic microparticles in xz-plane 38

311 Upward movement of the cluster 39312 Downward movement of the cluster 40

32 Position control of paramagnetic microparticles 40321 Motion control trials 40

33 Control of paramagnetic microparticle along a trajectory 42

4 Conclusions and Future work 4941 Conclusions 5042 Future work 50

Bibliography 51

List of Figures 61

List of Tables 67

Appendix 691 Communication protocol User Datagram Protocol (UDP) 69

1001 Sending bytes message from robot server to C 691002 Receiving byte message from C application

in the robot server 712 Creating a UDP communication in C application 723 Recieving a byte message using udp in java and C 734 Sending a byte message using udp in java and C 735 Image processing application example 73

vi

Acknowledgement

I would like to express my gratitude to my supervisor Dr Islam khalil for the usefulcomments remarks and engagement through the learning process of this master thesisFurthermore I would like to thank Dr Andre duffe for introducing me to the topic aswell for the support on the way

I would like to thank my parents and friends who have supported me throughout entireprocess both by keeping me harmonious and helping me putting pieces together Iwill be grateful forever for your love

1

Abstract

Magnetic control of drug carriers using systems with open-configurations is essentialto enable scaling to the size of in vivo applications In this study we demonstratemotion control of paramagnetic microparticles in a low Reynolds number fluid using apermanent magnet-based robotic system with an open-configuration The microparti-cles are controlled in three-dimensional (3D) space using a cylindrical NdFeB magnetthat is fixed to the end-effector of a robotic arm (Kuka iiwa r800 light weight robotwith 7 degrees of freedom and built in safety for better human interaction environ-ment) We develop a kinematic map between the position of the microparticles andthe configuration of the robotic arm and use this map as a basis of a closed-loop con-trol system based on the position of the microparticles Our closed-loop experimentalresults show the ability of the robot configuration to control the exerted magnetic fieldgradient on the dipole moment of the microparticles and achieve positioning in 3Dspace with maximum error of 300 microm and 600 microm in the steady-state during setpointand trajectory tracking respectively

3

1

Introduction

11 Motivation

Cancer is considered as the world second leading cause of death after heart disease [1]Survival rates are increasing owing to the improved medical treatment However thecurrent methods of treatment have negative side-effects on the body such as painloss of balance malnutrition and blood disorder Therefore much effort has beenexpanded to achieve targeted therapy that can treat cancer more safely than conven-tional methods [2] This work investigates the utlilzation of magnetic drug carriers(paramagnetic microparticles) that can reach the targeted position via a magnetic-based robotic system This study is necessary to translate the microparticles intoin vivo application

12 Electromagnetic Systems and Magnetic Control

Magnetically-driven and self-driven medical microrobots have the potential to deliverdrugs to deep seated regions in the human body Paramagnetic microparticles andhelical robots are examples of magnetically-driven microrobots while magnetotacticbacteria is a type of self-driven microrobots that have the potential to controllably

5

1 Introduction

Figure 11 The OctoMag prototype is designed and constructed at ETH Zurich Thesystem contains eight electromagnets in closed configuration [6]

achieve targeted drug delivery [3] A study was made for magnetic nanoparticles withsurface bound drug molecules type of microrobots were injected into the vascular sys-tem upstream from malignant tissue and were captured at the tumor using a localapplied magnetic field [4] Iron-oxide nanoparticles have been used in vitro by achiev-ing selective penetration of MCF-7 breast cancer cell without causing any damage tothe memberane and without any effect on the cell morphology [5] These differenttypes of microrobots are controlled wirelessly using controlled magnetic fields whichcan be generated using several methods The first method uses electromagnets thatare simultaneously position and current controlled The second method uses position-controlled permanent magnets Closed-configurations systems are generally thoughtto be essential for diverse biomedical applications during in vitro experimentation [7][8] The OctoMag shown in Fig 11 is a type of closed-loop control magnetic systemwith computer vision implemented for accurate positioning [2] [6] The OctoMag isdesigned for controlling the intraocular microrobots in minimally invasive retinal ther-apy and diagnosis It is also has been used as a wireless micromanipulation systemunder a light microscope Due to several technological barriers these systems cannotbe scaled up to the size of invivo applications [9] as it is generally not viable to achieverelatively high gradients at a distance They are placed in a way to form a closed-

6

12 Electromagnetic Systems and Magnetic Control

Figure 12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltechnolo-gie GmbH Rostock-Warnemuende Germany) in three-dimensional spaceThe magnetic system consists of 8 iron-core electromagnets that surroundsa water reservoir Magnetic-based robotic system give more freedom inmoving through trajectories compared to this system However this sys-tem showed greater positioning accuracy compared to the magnet-basedone [36]

configuration where the workspace containing microrobot which can be in the formof paramagnetic microparticles and nanoparticles The advantage of these particlesover the other types of microrobots is that they are biodegradable which give themthe opportunity to be used in most clinical applications However the magnetic forcesextended on these particles are small due to the dependency of the magnetic force onthe size and geometry of the object itself This problem can be solved by increasingthe directional derivatives in the applied field found between these coils (usually 4 or 8electromagentic coils) as shown Fig 12 The limited projection distance of the mag-netic field gradient constrains the workspace of these systems to tens of millimeters atmost

It is also difficult to integrate a clinical imaging modality to magnetic systems with

7

1 Introduction

closed configuration [10] [13] for instance These challenges have been partially over-come by Mahoney et al [9] [11] A combination of magnetic field-driven helicalmicrorobots [12] [15] and a robotic arm with a fixed permanent magnet to its endef-fector has been proposed and used to achieve locomotion throughout relatively largeworkspace in 3D space The rotating dipole field enables the microrobot to achievehelical propulsion while the robotic arm allows the rotating permanent magnet togenerate the rotating fields at any point within its workspace Recently Nelson etal [16] have utilized the non-uniformity of the rotating field in generating two inde-pendent rotating magnetic fields using a single magnet dipole An open configurationof two synchronized rotating dipole fields has been also introduced by Alshafeei etal to control the motion of helical microrobot inside a catheter segment [17] and in3D space [18] The open configuration of this magnetic-based robotic system enablesscaling to the size of in vivo applications and its ability to remove blood clots hasbeen demonstrated in vitro [19] An electromagnetic coil and a permanent magnethave been integrated to the end-effector of a robotic arm with four degrees-of-freedom(DOFs) in [20] and a comparative study has shown the ability of the coil to achievehigher positioning accuracy of paramagnetic microparticles compared to the perma-nent magnet Although electromagnetic coils provide greater positioning accuracyowing to the ability to control the magnitude of the field gradient via the currentinput it is desirable to benefit from the higher magnetic field and field gradient thatcan be generated using permanent magnets A theoretical analysis and experimen-tal tele-manipulation of trapped paramagnetic micro-particles inside gas bubbles wasintroduced by our research group Khalil et al by analyzing the interaction forces ofa trapped micro-particle in a gas bubble at the water-gas interface and estimatingthe interaction forces between the micro-particles and the bubble using a calibratedforce observer [22] Also another study was conducted for expermintly investigationthe non-contact manipulation of the microbeads using a cluster of microparticles Thenon-contact micromanipulation achieves successful releases by moving the microbeadstowards the reference positions [23]In this work we design a closed-loop control system to position clusters of param-agnetic microparticles in low-Reynolds number fluids The control is achieved in 3Dspace using a robotic arm with seven DOFs and a permanent magnet First we derivea mapping between the position of the cluster of paramagnetic microparticles and theconfiguration of the robotic arm This mapping is used as basis of a closed-loop controlsystem to position the cluster in 3D space

8

13 Thesis Structure

13 Thesis Structure

This thesis presents a control frame work to achieve positioning of drug carriers (para-magnetic magneticparticles) in 3D space using a magent-based robot in a closed-loopconfiguration The thesis is structured as follows

1 Chapter 2 describes the motion control and modeling of the paramagnetic mi-croparticles in 3D space

bull Section 21 presents the interface of the robotic arm (KUKA iiwa) thecommunication protocol used and the two sides of the interface in java andC

bull Section 22 explains the camera calibration to extract the internsic param-eters of the two cameras used in the experiments

bull Section 23 provides the image processing used to track the particle in real-time

bull Section 24 provides the kinematic model of the robotic arm and the mi-croparticles and the closed-loop control system design

2 Chapter 3 presents the experiments part showing the different positioning ofthe particles in the 3D space

bull Section 31 provides the control of a cluster of particles in xy-plane

bull Section 32 provides the control of a cluster of paricles in yz-plane

bull Section 33 provides the motion of the particles while following a line tra-jectory

bull Section 34 provides the motion of the particles while following a sinwave

3 Chapter 4 concludes and provides directions for future work

bull Section 41 provides the conclusions of this work

bull Section 42 discusses suggestions pertaining to our future work

9

1 Introduction

4 Chapter 5 presnsents the appendix and references used for this thesis

10

2

Experiments preparation

21 Modeling of a KUKA IIWA Serial-Link Manipulator

KUKA iiwa R800 (7 DOFs robot KUKA GmbH Germany) was used in this workand an interface was implemented to control the robot

211 Robots in Minimally Invasive Interventions

The range of applications for medical robots has increased in recent years Hospitalsreport hundreds and even thousands of procedures performed with the da Vinci robotshown in Fig21 which now has the delicate ability to precisely peel a grape It isused in all kinds of surgeries from heart-valve repair to tumor removal and acrossvarious surgical disciplines The ROBODOC shown in Fig22 now combines withORTHODOC to preplan surgical procedures and carry them out with relatively highaccuracy ROBODOC also assists the physicians with knee replacements Roboticprosthetics now include sensors that are attached to the muscles and nerves allowingpatients to experience a sense of touch and to control movements with thought Pros-thetic hands allow fine motor skills such as writing typing or playing piano Robotsmight have feelings after all with the help of a robotic exoskeleton stroke victimscan regain lost arm movement quicker than with traditional physical therapy Able to

11

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

Contents

Acknowledgement 1

Abstract 3

1 Introduction 511 Motivation 512 Electromagnetic Systems and Magnetic Control 513 Thesis Structure 9

2 Experiments preparation 1121 Modeling of a KUKA IIWA Serial-Link Manipulator 11

211 Robots in Minimally Invasive Interventions 11212 KUKA iiwa r800 12213 Interface of the KUKA robot 12

2131 Communication protocols for connecting the robot serverwith a c application 14

214 Additional features added to the robot arm to allow a free mo-tion in the null frame and motion to a target position 15

21401 Gravity compensation feature of allowing robotarm motion in the null space 16

21402 Motion feature for driving the robot arm toan aimed position 16

22 Camera calibration to extract the intrinsic parameters 19221 Camera modelling 20

23 Feature Tracking of Microparticles 2724 Kinematic Modeling of the Robotic Arm and Microparticle 2925 Control of the Magnetic-Based Robotic System 34

v

Contents

3 Experiments 3731 Position control of paramagnetic microparticles in xz-plane 38

311 Upward movement of the cluster 39312 Downward movement of the cluster 40

32 Position control of paramagnetic microparticles 40321 Motion control trials 40

33 Control of paramagnetic microparticle along a trajectory 42

4 Conclusions and Future work 4941 Conclusions 5042 Future work 50

Bibliography 51

List of Figures 61

List of Tables 67

Appendix 691 Communication protocol User Datagram Protocol (UDP) 69

1001 Sending bytes message from robot server to C 691002 Receiving byte message from C application

in the robot server 712 Creating a UDP communication in C application 723 Recieving a byte message using udp in java and C 734 Sending a byte message using udp in java and C 735 Image processing application example 73

vi

Acknowledgement

I would like to express my gratitude to my supervisor Dr Islam khalil for the usefulcomments remarks and engagement through the learning process of this master thesisFurthermore I would like to thank Dr Andre duffe for introducing me to the topic aswell for the support on the way

I would like to thank my parents and friends who have supported me throughout entireprocess both by keeping me harmonious and helping me putting pieces together Iwill be grateful forever for your love

1

Abstract

Magnetic control of drug carriers using systems with open-configurations is essentialto enable scaling to the size of in vivo applications In this study we demonstratemotion control of paramagnetic microparticles in a low Reynolds number fluid using apermanent magnet-based robotic system with an open-configuration The microparti-cles are controlled in three-dimensional (3D) space using a cylindrical NdFeB magnetthat is fixed to the end-effector of a robotic arm (Kuka iiwa r800 light weight robotwith 7 degrees of freedom and built in safety for better human interaction environ-ment) We develop a kinematic map between the position of the microparticles andthe configuration of the robotic arm and use this map as a basis of a closed-loop con-trol system based on the position of the microparticles Our closed-loop experimentalresults show the ability of the robot configuration to control the exerted magnetic fieldgradient on the dipole moment of the microparticles and achieve positioning in 3Dspace with maximum error of 300 microm and 600 microm in the steady-state during setpointand trajectory tracking respectively

3

1

Introduction

11 Motivation

Cancer is considered as the world second leading cause of death after heart disease [1]Survival rates are increasing owing to the improved medical treatment However thecurrent methods of treatment have negative side-effects on the body such as painloss of balance malnutrition and blood disorder Therefore much effort has beenexpanded to achieve targeted therapy that can treat cancer more safely than conven-tional methods [2] This work investigates the utlilzation of magnetic drug carriers(paramagnetic microparticles) that can reach the targeted position via a magnetic-based robotic system This study is necessary to translate the microparticles intoin vivo application

12 Electromagnetic Systems and Magnetic Control

Magnetically-driven and self-driven medical microrobots have the potential to deliverdrugs to deep seated regions in the human body Paramagnetic microparticles andhelical robots are examples of magnetically-driven microrobots while magnetotacticbacteria is a type of self-driven microrobots that have the potential to controllably

5

1 Introduction

Figure 11 The OctoMag prototype is designed and constructed at ETH Zurich Thesystem contains eight electromagnets in closed configuration [6]

achieve targeted drug delivery [3] A study was made for magnetic nanoparticles withsurface bound drug molecules type of microrobots were injected into the vascular sys-tem upstream from malignant tissue and were captured at the tumor using a localapplied magnetic field [4] Iron-oxide nanoparticles have been used in vitro by achiev-ing selective penetration of MCF-7 breast cancer cell without causing any damage tothe memberane and without any effect on the cell morphology [5] These differenttypes of microrobots are controlled wirelessly using controlled magnetic fields whichcan be generated using several methods The first method uses electromagnets thatare simultaneously position and current controlled The second method uses position-controlled permanent magnets Closed-configurations systems are generally thoughtto be essential for diverse biomedical applications during in vitro experimentation [7][8] The OctoMag shown in Fig 11 is a type of closed-loop control magnetic systemwith computer vision implemented for accurate positioning [2] [6] The OctoMag isdesigned for controlling the intraocular microrobots in minimally invasive retinal ther-apy and diagnosis It is also has been used as a wireless micromanipulation systemunder a light microscope Due to several technological barriers these systems cannotbe scaled up to the size of invivo applications [9] as it is generally not viable to achieverelatively high gradients at a distance They are placed in a way to form a closed-

6

12 Electromagnetic Systems and Magnetic Control

Figure 12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltechnolo-gie GmbH Rostock-Warnemuende Germany) in three-dimensional spaceThe magnetic system consists of 8 iron-core electromagnets that surroundsa water reservoir Magnetic-based robotic system give more freedom inmoving through trajectories compared to this system However this sys-tem showed greater positioning accuracy compared to the magnet-basedone [36]

configuration where the workspace containing microrobot which can be in the formof paramagnetic microparticles and nanoparticles The advantage of these particlesover the other types of microrobots is that they are biodegradable which give themthe opportunity to be used in most clinical applications However the magnetic forcesextended on these particles are small due to the dependency of the magnetic force onthe size and geometry of the object itself This problem can be solved by increasingthe directional derivatives in the applied field found between these coils (usually 4 or 8electromagentic coils) as shown Fig 12 The limited projection distance of the mag-netic field gradient constrains the workspace of these systems to tens of millimeters atmost

It is also difficult to integrate a clinical imaging modality to magnetic systems with

7

1 Introduction

closed configuration [10] [13] for instance These challenges have been partially over-come by Mahoney et al [9] [11] A combination of magnetic field-driven helicalmicrorobots [12] [15] and a robotic arm with a fixed permanent magnet to its endef-fector has been proposed and used to achieve locomotion throughout relatively largeworkspace in 3D space The rotating dipole field enables the microrobot to achievehelical propulsion while the robotic arm allows the rotating permanent magnet togenerate the rotating fields at any point within its workspace Recently Nelson etal [16] have utilized the non-uniformity of the rotating field in generating two inde-pendent rotating magnetic fields using a single magnet dipole An open configurationof two synchronized rotating dipole fields has been also introduced by Alshafeei etal to control the motion of helical microrobot inside a catheter segment [17] and in3D space [18] The open configuration of this magnetic-based robotic system enablesscaling to the size of in vivo applications and its ability to remove blood clots hasbeen demonstrated in vitro [19] An electromagnetic coil and a permanent magnethave been integrated to the end-effector of a robotic arm with four degrees-of-freedom(DOFs) in [20] and a comparative study has shown the ability of the coil to achievehigher positioning accuracy of paramagnetic microparticles compared to the perma-nent magnet Although electromagnetic coils provide greater positioning accuracyowing to the ability to control the magnitude of the field gradient via the currentinput it is desirable to benefit from the higher magnetic field and field gradient thatcan be generated using permanent magnets A theoretical analysis and experimen-tal tele-manipulation of trapped paramagnetic micro-particles inside gas bubbles wasintroduced by our research group Khalil et al by analyzing the interaction forces ofa trapped micro-particle in a gas bubble at the water-gas interface and estimatingthe interaction forces between the micro-particles and the bubble using a calibratedforce observer [22] Also another study was conducted for expermintly investigationthe non-contact manipulation of the microbeads using a cluster of microparticles Thenon-contact micromanipulation achieves successful releases by moving the microbeadstowards the reference positions [23]In this work we design a closed-loop control system to position clusters of param-agnetic microparticles in low-Reynolds number fluids The control is achieved in 3Dspace using a robotic arm with seven DOFs and a permanent magnet First we derivea mapping between the position of the cluster of paramagnetic microparticles and theconfiguration of the robotic arm This mapping is used as basis of a closed-loop controlsystem to position the cluster in 3D space

8

13 Thesis Structure

13 Thesis Structure

This thesis presents a control frame work to achieve positioning of drug carriers (para-magnetic magneticparticles) in 3D space using a magent-based robot in a closed-loopconfiguration The thesis is structured as follows

1 Chapter 2 describes the motion control and modeling of the paramagnetic mi-croparticles in 3D space

bull Section 21 presents the interface of the robotic arm (KUKA iiwa) thecommunication protocol used and the two sides of the interface in java andC

bull Section 22 explains the camera calibration to extract the internsic param-eters of the two cameras used in the experiments

bull Section 23 provides the image processing used to track the particle in real-time

bull Section 24 provides the kinematic model of the robotic arm and the mi-croparticles and the closed-loop control system design

2 Chapter 3 presents the experiments part showing the different positioning ofthe particles in the 3D space

bull Section 31 provides the control of a cluster of particles in xy-plane

bull Section 32 provides the control of a cluster of paricles in yz-plane

bull Section 33 provides the motion of the particles while following a line tra-jectory

bull Section 34 provides the motion of the particles while following a sinwave

3 Chapter 4 concludes and provides directions for future work

bull Section 41 provides the conclusions of this work

bull Section 42 discusses suggestions pertaining to our future work

9

1 Introduction

4 Chapter 5 presnsents the appendix and references used for this thesis

10

2

Experiments preparation

21 Modeling of a KUKA IIWA Serial-Link Manipulator

KUKA iiwa R800 (7 DOFs robot KUKA GmbH Germany) was used in this workand an interface was implemented to control the robot

211 Robots in Minimally Invasive Interventions

The range of applications for medical robots has increased in recent years Hospitalsreport hundreds and even thousands of procedures performed with the da Vinci robotshown in Fig21 which now has the delicate ability to precisely peel a grape It isused in all kinds of surgeries from heart-valve repair to tumor removal and acrossvarious surgical disciplines The ROBODOC shown in Fig22 now combines withORTHODOC to preplan surgical procedures and carry them out with relatively highaccuracy ROBODOC also assists the physicians with knee replacements Roboticprosthetics now include sensors that are attached to the muscles and nerves allowingpatients to experience a sense of touch and to control movements with thought Pros-thetic hands allow fine motor skills such as writing typing or playing piano Robotsmight have feelings after all with the help of a robotic exoskeleton stroke victimscan regain lost arm movement quicker than with traditional physical therapy Able to

11

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

Contents

3 Experiments 3731 Position control of paramagnetic microparticles in xz-plane 38

311 Upward movement of the cluster 39312 Downward movement of the cluster 40

32 Position control of paramagnetic microparticles 40321 Motion control trials 40

33 Control of paramagnetic microparticle along a trajectory 42

4 Conclusions and Future work 4941 Conclusions 5042 Future work 50

Bibliography 51

List of Figures 61

List of Tables 67

Appendix 691 Communication protocol User Datagram Protocol (UDP) 69

1001 Sending bytes message from robot server to C 691002 Receiving byte message from C application

in the robot server 712 Creating a UDP communication in C application 723 Recieving a byte message using udp in java and C 734 Sending a byte message using udp in java and C 735 Image processing application example 73

vi

Acknowledgement

I would like to express my gratitude to my supervisor Dr Islam khalil for the usefulcomments remarks and engagement through the learning process of this master thesisFurthermore I would like to thank Dr Andre duffe for introducing me to the topic aswell for the support on the way

I would like to thank my parents and friends who have supported me throughout entireprocess both by keeping me harmonious and helping me putting pieces together Iwill be grateful forever for your love

1

Abstract

Magnetic control of drug carriers using systems with open-configurations is essentialto enable scaling to the size of in vivo applications In this study we demonstratemotion control of paramagnetic microparticles in a low Reynolds number fluid using apermanent magnet-based robotic system with an open-configuration The microparti-cles are controlled in three-dimensional (3D) space using a cylindrical NdFeB magnetthat is fixed to the end-effector of a robotic arm (Kuka iiwa r800 light weight robotwith 7 degrees of freedom and built in safety for better human interaction environ-ment) We develop a kinematic map between the position of the microparticles andthe configuration of the robotic arm and use this map as a basis of a closed-loop con-trol system based on the position of the microparticles Our closed-loop experimentalresults show the ability of the robot configuration to control the exerted magnetic fieldgradient on the dipole moment of the microparticles and achieve positioning in 3Dspace with maximum error of 300 microm and 600 microm in the steady-state during setpointand trajectory tracking respectively

3

1

Introduction

11 Motivation

Cancer is considered as the world second leading cause of death after heart disease [1]Survival rates are increasing owing to the improved medical treatment However thecurrent methods of treatment have negative side-effects on the body such as painloss of balance malnutrition and blood disorder Therefore much effort has beenexpanded to achieve targeted therapy that can treat cancer more safely than conven-tional methods [2] This work investigates the utlilzation of magnetic drug carriers(paramagnetic microparticles) that can reach the targeted position via a magnetic-based robotic system This study is necessary to translate the microparticles intoin vivo application

12 Electromagnetic Systems and Magnetic Control

Magnetically-driven and self-driven medical microrobots have the potential to deliverdrugs to deep seated regions in the human body Paramagnetic microparticles andhelical robots are examples of magnetically-driven microrobots while magnetotacticbacteria is a type of self-driven microrobots that have the potential to controllably

5

1 Introduction

Figure 11 The OctoMag prototype is designed and constructed at ETH Zurich Thesystem contains eight electromagnets in closed configuration [6]

achieve targeted drug delivery [3] A study was made for magnetic nanoparticles withsurface bound drug molecules type of microrobots were injected into the vascular sys-tem upstream from malignant tissue and were captured at the tumor using a localapplied magnetic field [4] Iron-oxide nanoparticles have been used in vitro by achiev-ing selective penetration of MCF-7 breast cancer cell without causing any damage tothe memberane and without any effect on the cell morphology [5] These differenttypes of microrobots are controlled wirelessly using controlled magnetic fields whichcan be generated using several methods The first method uses electromagnets thatare simultaneously position and current controlled The second method uses position-controlled permanent magnets Closed-configurations systems are generally thoughtto be essential for diverse biomedical applications during in vitro experimentation [7][8] The OctoMag shown in Fig 11 is a type of closed-loop control magnetic systemwith computer vision implemented for accurate positioning [2] [6] The OctoMag isdesigned for controlling the intraocular microrobots in minimally invasive retinal ther-apy and diagnosis It is also has been used as a wireless micromanipulation systemunder a light microscope Due to several technological barriers these systems cannotbe scaled up to the size of invivo applications [9] as it is generally not viable to achieverelatively high gradients at a distance They are placed in a way to form a closed-

6

12 Electromagnetic Systems and Magnetic Control

Figure 12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltechnolo-gie GmbH Rostock-Warnemuende Germany) in three-dimensional spaceThe magnetic system consists of 8 iron-core electromagnets that surroundsa water reservoir Magnetic-based robotic system give more freedom inmoving through trajectories compared to this system However this sys-tem showed greater positioning accuracy compared to the magnet-basedone [36]

configuration where the workspace containing microrobot which can be in the formof paramagnetic microparticles and nanoparticles The advantage of these particlesover the other types of microrobots is that they are biodegradable which give themthe opportunity to be used in most clinical applications However the magnetic forcesextended on these particles are small due to the dependency of the magnetic force onthe size and geometry of the object itself This problem can be solved by increasingthe directional derivatives in the applied field found between these coils (usually 4 or 8electromagentic coils) as shown Fig 12 The limited projection distance of the mag-netic field gradient constrains the workspace of these systems to tens of millimeters atmost

It is also difficult to integrate a clinical imaging modality to magnetic systems with

7

1 Introduction

closed configuration [10] [13] for instance These challenges have been partially over-come by Mahoney et al [9] [11] A combination of magnetic field-driven helicalmicrorobots [12] [15] and a robotic arm with a fixed permanent magnet to its endef-fector has been proposed and used to achieve locomotion throughout relatively largeworkspace in 3D space The rotating dipole field enables the microrobot to achievehelical propulsion while the robotic arm allows the rotating permanent magnet togenerate the rotating fields at any point within its workspace Recently Nelson etal [16] have utilized the non-uniformity of the rotating field in generating two inde-pendent rotating magnetic fields using a single magnet dipole An open configurationof two synchronized rotating dipole fields has been also introduced by Alshafeei etal to control the motion of helical microrobot inside a catheter segment [17] and in3D space [18] The open configuration of this magnetic-based robotic system enablesscaling to the size of in vivo applications and its ability to remove blood clots hasbeen demonstrated in vitro [19] An electromagnetic coil and a permanent magnethave been integrated to the end-effector of a robotic arm with four degrees-of-freedom(DOFs) in [20] and a comparative study has shown the ability of the coil to achievehigher positioning accuracy of paramagnetic microparticles compared to the perma-nent magnet Although electromagnetic coils provide greater positioning accuracyowing to the ability to control the magnitude of the field gradient via the currentinput it is desirable to benefit from the higher magnetic field and field gradient thatcan be generated using permanent magnets A theoretical analysis and experimen-tal tele-manipulation of trapped paramagnetic micro-particles inside gas bubbles wasintroduced by our research group Khalil et al by analyzing the interaction forces ofa trapped micro-particle in a gas bubble at the water-gas interface and estimatingthe interaction forces between the micro-particles and the bubble using a calibratedforce observer [22] Also another study was conducted for expermintly investigationthe non-contact manipulation of the microbeads using a cluster of microparticles Thenon-contact micromanipulation achieves successful releases by moving the microbeadstowards the reference positions [23]In this work we design a closed-loop control system to position clusters of param-agnetic microparticles in low-Reynolds number fluids The control is achieved in 3Dspace using a robotic arm with seven DOFs and a permanent magnet First we derivea mapping between the position of the cluster of paramagnetic microparticles and theconfiguration of the robotic arm This mapping is used as basis of a closed-loop controlsystem to position the cluster in 3D space

8

13 Thesis Structure

13 Thesis Structure

This thesis presents a control frame work to achieve positioning of drug carriers (para-magnetic magneticparticles) in 3D space using a magent-based robot in a closed-loopconfiguration The thesis is structured as follows

1 Chapter 2 describes the motion control and modeling of the paramagnetic mi-croparticles in 3D space

bull Section 21 presents the interface of the robotic arm (KUKA iiwa) thecommunication protocol used and the two sides of the interface in java andC

bull Section 22 explains the camera calibration to extract the internsic param-eters of the two cameras used in the experiments

bull Section 23 provides the image processing used to track the particle in real-time

bull Section 24 provides the kinematic model of the robotic arm and the mi-croparticles and the closed-loop control system design

2 Chapter 3 presents the experiments part showing the different positioning ofthe particles in the 3D space

bull Section 31 provides the control of a cluster of particles in xy-plane

bull Section 32 provides the control of a cluster of paricles in yz-plane

bull Section 33 provides the motion of the particles while following a line tra-jectory

bull Section 34 provides the motion of the particles while following a sinwave

3 Chapter 4 concludes and provides directions for future work

bull Section 41 provides the conclusions of this work

bull Section 42 discusses suggestions pertaining to our future work

9

1 Introduction

4 Chapter 5 presnsents the appendix and references used for this thesis

10

2

Experiments preparation

21 Modeling of a KUKA IIWA Serial-Link Manipulator

KUKA iiwa R800 (7 DOFs robot KUKA GmbH Germany) was used in this workand an interface was implemented to control the robot

211 Robots in Minimally Invasive Interventions

The range of applications for medical robots has increased in recent years Hospitalsreport hundreds and even thousands of procedures performed with the da Vinci robotshown in Fig21 which now has the delicate ability to precisely peel a grape It isused in all kinds of surgeries from heart-valve repair to tumor removal and acrossvarious surgical disciplines The ROBODOC shown in Fig22 now combines withORTHODOC to preplan surgical procedures and carry them out with relatively highaccuracy ROBODOC also assists the physicians with knee replacements Roboticprosthetics now include sensors that are attached to the muscles and nerves allowingpatients to experience a sense of touch and to control movements with thought Pros-thetic hands allow fine motor skills such as writing typing or playing piano Robotsmight have feelings after all with the help of a robotic exoskeleton stroke victimscan regain lost arm movement quicker than with traditional physical therapy Able to

11

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

Acknowledgement

I would like to express my gratitude to my supervisor Dr Islam khalil for the usefulcomments remarks and engagement through the learning process of this master thesisFurthermore I would like to thank Dr Andre duffe for introducing me to the topic aswell for the support on the way

I would like to thank my parents and friends who have supported me throughout entireprocess both by keeping me harmonious and helping me putting pieces together Iwill be grateful forever for your love

1

Abstract

Magnetic control of drug carriers using systems with open-configurations is essentialto enable scaling to the size of in vivo applications In this study we demonstratemotion control of paramagnetic microparticles in a low Reynolds number fluid using apermanent magnet-based robotic system with an open-configuration The microparti-cles are controlled in three-dimensional (3D) space using a cylindrical NdFeB magnetthat is fixed to the end-effector of a robotic arm (Kuka iiwa r800 light weight robotwith 7 degrees of freedom and built in safety for better human interaction environ-ment) We develop a kinematic map between the position of the microparticles andthe configuration of the robotic arm and use this map as a basis of a closed-loop con-trol system based on the position of the microparticles Our closed-loop experimentalresults show the ability of the robot configuration to control the exerted magnetic fieldgradient on the dipole moment of the microparticles and achieve positioning in 3Dspace with maximum error of 300 microm and 600 microm in the steady-state during setpointand trajectory tracking respectively

3

1

Introduction

11 Motivation

Cancer is considered as the world second leading cause of death after heart disease [1]Survival rates are increasing owing to the improved medical treatment However thecurrent methods of treatment have negative side-effects on the body such as painloss of balance malnutrition and blood disorder Therefore much effort has beenexpanded to achieve targeted therapy that can treat cancer more safely than conven-tional methods [2] This work investigates the utlilzation of magnetic drug carriers(paramagnetic microparticles) that can reach the targeted position via a magnetic-based robotic system This study is necessary to translate the microparticles intoin vivo application

12 Electromagnetic Systems and Magnetic Control

Magnetically-driven and self-driven medical microrobots have the potential to deliverdrugs to deep seated regions in the human body Paramagnetic microparticles andhelical robots are examples of magnetically-driven microrobots while magnetotacticbacteria is a type of self-driven microrobots that have the potential to controllably

5

1 Introduction

Figure 11 The OctoMag prototype is designed and constructed at ETH Zurich Thesystem contains eight electromagnets in closed configuration [6]

achieve targeted drug delivery [3] A study was made for magnetic nanoparticles withsurface bound drug molecules type of microrobots were injected into the vascular sys-tem upstream from malignant tissue and were captured at the tumor using a localapplied magnetic field [4] Iron-oxide nanoparticles have been used in vitro by achiev-ing selective penetration of MCF-7 breast cancer cell without causing any damage tothe memberane and without any effect on the cell morphology [5] These differenttypes of microrobots are controlled wirelessly using controlled magnetic fields whichcan be generated using several methods The first method uses electromagnets thatare simultaneously position and current controlled The second method uses position-controlled permanent magnets Closed-configurations systems are generally thoughtto be essential for diverse biomedical applications during in vitro experimentation [7][8] The OctoMag shown in Fig 11 is a type of closed-loop control magnetic systemwith computer vision implemented for accurate positioning [2] [6] The OctoMag isdesigned for controlling the intraocular microrobots in minimally invasive retinal ther-apy and diagnosis It is also has been used as a wireless micromanipulation systemunder a light microscope Due to several technological barriers these systems cannotbe scaled up to the size of invivo applications [9] as it is generally not viable to achieverelatively high gradients at a distance They are placed in a way to form a closed-

6

12 Electromagnetic Systems and Magnetic Control

Figure 12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltechnolo-gie GmbH Rostock-Warnemuende Germany) in three-dimensional spaceThe magnetic system consists of 8 iron-core electromagnets that surroundsa water reservoir Magnetic-based robotic system give more freedom inmoving through trajectories compared to this system However this sys-tem showed greater positioning accuracy compared to the magnet-basedone [36]

configuration where the workspace containing microrobot which can be in the formof paramagnetic microparticles and nanoparticles The advantage of these particlesover the other types of microrobots is that they are biodegradable which give themthe opportunity to be used in most clinical applications However the magnetic forcesextended on these particles are small due to the dependency of the magnetic force onthe size and geometry of the object itself This problem can be solved by increasingthe directional derivatives in the applied field found between these coils (usually 4 or 8electromagentic coils) as shown Fig 12 The limited projection distance of the mag-netic field gradient constrains the workspace of these systems to tens of millimeters atmost

It is also difficult to integrate a clinical imaging modality to magnetic systems with

7

1 Introduction

closed configuration [10] [13] for instance These challenges have been partially over-come by Mahoney et al [9] [11] A combination of magnetic field-driven helicalmicrorobots [12] [15] and a robotic arm with a fixed permanent magnet to its endef-fector has been proposed and used to achieve locomotion throughout relatively largeworkspace in 3D space The rotating dipole field enables the microrobot to achievehelical propulsion while the robotic arm allows the rotating permanent magnet togenerate the rotating fields at any point within its workspace Recently Nelson etal [16] have utilized the non-uniformity of the rotating field in generating two inde-pendent rotating magnetic fields using a single magnet dipole An open configurationof two synchronized rotating dipole fields has been also introduced by Alshafeei etal to control the motion of helical microrobot inside a catheter segment [17] and in3D space [18] The open configuration of this magnetic-based robotic system enablesscaling to the size of in vivo applications and its ability to remove blood clots hasbeen demonstrated in vitro [19] An electromagnetic coil and a permanent magnethave been integrated to the end-effector of a robotic arm with four degrees-of-freedom(DOFs) in [20] and a comparative study has shown the ability of the coil to achievehigher positioning accuracy of paramagnetic microparticles compared to the perma-nent magnet Although electromagnetic coils provide greater positioning accuracyowing to the ability to control the magnitude of the field gradient via the currentinput it is desirable to benefit from the higher magnetic field and field gradient thatcan be generated using permanent magnets A theoretical analysis and experimen-tal tele-manipulation of trapped paramagnetic micro-particles inside gas bubbles wasintroduced by our research group Khalil et al by analyzing the interaction forces ofa trapped micro-particle in a gas bubble at the water-gas interface and estimatingthe interaction forces between the micro-particles and the bubble using a calibratedforce observer [22] Also another study was conducted for expermintly investigationthe non-contact manipulation of the microbeads using a cluster of microparticles Thenon-contact micromanipulation achieves successful releases by moving the microbeadstowards the reference positions [23]In this work we design a closed-loop control system to position clusters of param-agnetic microparticles in low-Reynolds number fluids The control is achieved in 3Dspace using a robotic arm with seven DOFs and a permanent magnet First we derivea mapping between the position of the cluster of paramagnetic microparticles and theconfiguration of the robotic arm This mapping is used as basis of a closed-loop controlsystem to position the cluster in 3D space

8

13 Thesis Structure

13 Thesis Structure

This thesis presents a control frame work to achieve positioning of drug carriers (para-magnetic magneticparticles) in 3D space using a magent-based robot in a closed-loopconfiguration The thesis is structured as follows

1 Chapter 2 describes the motion control and modeling of the paramagnetic mi-croparticles in 3D space

bull Section 21 presents the interface of the robotic arm (KUKA iiwa) thecommunication protocol used and the two sides of the interface in java andC

bull Section 22 explains the camera calibration to extract the internsic param-eters of the two cameras used in the experiments

bull Section 23 provides the image processing used to track the particle in real-time

bull Section 24 provides the kinematic model of the robotic arm and the mi-croparticles and the closed-loop control system design

2 Chapter 3 presents the experiments part showing the different positioning ofthe particles in the 3D space

bull Section 31 provides the control of a cluster of particles in xy-plane

bull Section 32 provides the control of a cluster of paricles in yz-plane

bull Section 33 provides the motion of the particles while following a line tra-jectory

bull Section 34 provides the motion of the particles while following a sinwave

3 Chapter 4 concludes and provides directions for future work

bull Section 41 provides the conclusions of this work

bull Section 42 discusses suggestions pertaining to our future work

9

1 Introduction

4 Chapter 5 presnsents the appendix and references used for this thesis

10

2

Experiments preparation

21 Modeling of a KUKA IIWA Serial-Link Manipulator

KUKA iiwa R800 (7 DOFs robot KUKA GmbH Germany) was used in this workand an interface was implemented to control the robot

211 Robots in Minimally Invasive Interventions

The range of applications for medical robots has increased in recent years Hospitalsreport hundreds and even thousands of procedures performed with the da Vinci robotshown in Fig21 which now has the delicate ability to precisely peel a grape It isused in all kinds of surgeries from heart-valve repair to tumor removal and acrossvarious surgical disciplines The ROBODOC shown in Fig22 now combines withORTHODOC to preplan surgical procedures and carry them out with relatively highaccuracy ROBODOC also assists the physicians with knee replacements Roboticprosthetics now include sensors that are attached to the muscles and nerves allowingpatients to experience a sense of touch and to control movements with thought Pros-thetic hands allow fine motor skills such as writing typing or playing piano Robotsmight have feelings after all with the help of a robotic exoskeleton stroke victimscan regain lost arm movement quicker than with traditional physical therapy Able to

11

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

Abstract

Magnetic control of drug carriers using systems with open-configurations is essentialto enable scaling to the size of in vivo applications In this study we demonstratemotion control of paramagnetic microparticles in a low Reynolds number fluid using apermanent magnet-based robotic system with an open-configuration The microparti-cles are controlled in three-dimensional (3D) space using a cylindrical NdFeB magnetthat is fixed to the end-effector of a robotic arm (Kuka iiwa r800 light weight robotwith 7 degrees of freedom and built in safety for better human interaction environ-ment) We develop a kinematic map between the position of the microparticles andthe configuration of the robotic arm and use this map as a basis of a closed-loop con-trol system based on the position of the microparticles Our closed-loop experimentalresults show the ability of the robot configuration to control the exerted magnetic fieldgradient on the dipole moment of the microparticles and achieve positioning in 3Dspace with maximum error of 300 microm and 600 microm in the steady-state during setpointand trajectory tracking respectively

3

1

Introduction

11 Motivation

Cancer is considered as the world second leading cause of death after heart disease [1]Survival rates are increasing owing to the improved medical treatment However thecurrent methods of treatment have negative side-effects on the body such as painloss of balance malnutrition and blood disorder Therefore much effort has beenexpanded to achieve targeted therapy that can treat cancer more safely than conven-tional methods [2] This work investigates the utlilzation of magnetic drug carriers(paramagnetic microparticles) that can reach the targeted position via a magnetic-based robotic system This study is necessary to translate the microparticles intoin vivo application

12 Electromagnetic Systems and Magnetic Control

Magnetically-driven and self-driven medical microrobots have the potential to deliverdrugs to deep seated regions in the human body Paramagnetic microparticles andhelical robots are examples of magnetically-driven microrobots while magnetotacticbacteria is a type of self-driven microrobots that have the potential to controllably

5

1 Introduction

Figure 11 The OctoMag prototype is designed and constructed at ETH Zurich Thesystem contains eight electromagnets in closed configuration [6]

achieve targeted drug delivery [3] A study was made for magnetic nanoparticles withsurface bound drug molecules type of microrobots were injected into the vascular sys-tem upstream from malignant tissue and were captured at the tumor using a localapplied magnetic field [4] Iron-oxide nanoparticles have been used in vitro by achiev-ing selective penetration of MCF-7 breast cancer cell without causing any damage tothe memberane and without any effect on the cell morphology [5] These differenttypes of microrobots are controlled wirelessly using controlled magnetic fields whichcan be generated using several methods The first method uses electromagnets thatare simultaneously position and current controlled The second method uses position-controlled permanent magnets Closed-configurations systems are generally thoughtto be essential for diverse biomedical applications during in vitro experimentation [7][8] The OctoMag shown in Fig 11 is a type of closed-loop control magnetic systemwith computer vision implemented for accurate positioning [2] [6] The OctoMag isdesigned for controlling the intraocular microrobots in minimally invasive retinal ther-apy and diagnosis It is also has been used as a wireless micromanipulation systemunder a light microscope Due to several technological barriers these systems cannotbe scaled up to the size of invivo applications [9] as it is generally not viable to achieverelatively high gradients at a distance They are placed in a way to form a closed-

6

12 Electromagnetic Systems and Magnetic Control

Figure 12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltechnolo-gie GmbH Rostock-Warnemuende Germany) in three-dimensional spaceThe magnetic system consists of 8 iron-core electromagnets that surroundsa water reservoir Magnetic-based robotic system give more freedom inmoving through trajectories compared to this system However this sys-tem showed greater positioning accuracy compared to the magnet-basedone [36]

configuration where the workspace containing microrobot which can be in the formof paramagnetic microparticles and nanoparticles The advantage of these particlesover the other types of microrobots is that they are biodegradable which give themthe opportunity to be used in most clinical applications However the magnetic forcesextended on these particles are small due to the dependency of the magnetic force onthe size and geometry of the object itself This problem can be solved by increasingthe directional derivatives in the applied field found between these coils (usually 4 or 8electromagentic coils) as shown Fig 12 The limited projection distance of the mag-netic field gradient constrains the workspace of these systems to tens of millimeters atmost

It is also difficult to integrate a clinical imaging modality to magnetic systems with

7

1 Introduction

closed configuration [10] [13] for instance These challenges have been partially over-come by Mahoney et al [9] [11] A combination of magnetic field-driven helicalmicrorobots [12] [15] and a robotic arm with a fixed permanent magnet to its endef-fector has been proposed and used to achieve locomotion throughout relatively largeworkspace in 3D space The rotating dipole field enables the microrobot to achievehelical propulsion while the robotic arm allows the rotating permanent magnet togenerate the rotating fields at any point within its workspace Recently Nelson etal [16] have utilized the non-uniformity of the rotating field in generating two inde-pendent rotating magnetic fields using a single magnet dipole An open configurationof two synchronized rotating dipole fields has been also introduced by Alshafeei etal to control the motion of helical microrobot inside a catheter segment [17] and in3D space [18] The open configuration of this magnetic-based robotic system enablesscaling to the size of in vivo applications and its ability to remove blood clots hasbeen demonstrated in vitro [19] An electromagnetic coil and a permanent magnethave been integrated to the end-effector of a robotic arm with four degrees-of-freedom(DOFs) in [20] and a comparative study has shown the ability of the coil to achievehigher positioning accuracy of paramagnetic microparticles compared to the perma-nent magnet Although electromagnetic coils provide greater positioning accuracyowing to the ability to control the magnitude of the field gradient via the currentinput it is desirable to benefit from the higher magnetic field and field gradient thatcan be generated using permanent magnets A theoretical analysis and experimen-tal tele-manipulation of trapped paramagnetic micro-particles inside gas bubbles wasintroduced by our research group Khalil et al by analyzing the interaction forces ofa trapped micro-particle in a gas bubble at the water-gas interface and estimatingthe interaction forces between the micro-particles and the bubble using a calibratedforce observer [22] Also another study was conducted for expermintly investigationthe non-contact manipulation of the microbeads using a cluster of microparticles Thenon-contact micromanipulation achieves successful releases by moving the microbeadstowards the reference positions [23]In this work we design a closed-loop control system to position clusters of param-agnetic microparticles in low-Reynolds number fluids The control is achieved in 3Dspace using a robotic arm with seven DOFs and a permanent magnet First we derivea mapping between the position of the cluster of paramagnetic microparticles and theconfiguration of the robotic arm This mapping is used as basis of a closed-loop controlsystem to position the cluster in 3D space

8

13 Thesis Structure

13 Thesis Structure

This thesis presents a control frame work to achieve positioning of drug carriers (para-magnetic magneticparticles) in 3D space using a magent-based robot in a closed-loopconfiguration The thesis is structured as follows

1 Chapter 2 describes the motion control and modeling of the paramagnetic mi-croparticles in 3D space

bull Section 21 presents the interface of the robotic arm (KUKA iiwa) thecommunication protocol used and the two sides of the interface in java andC

bull Section 22 explains the camera calibration to extract the internsic param-eters of the two cameras used in the experiments

bull Section 23 provides the image processing used to track the particle in real-time

bull Section 24 provides the kinematic model of the robotic arm and the mi-croparticles and the closed-loop control system design

2 Chapter 3 presents the experiments part showing the different positioning ofthe particles in the 3D space

bull Section 31 provides the control of a cluster of particles in xy-plane

bull Section 32 provides the control of a cluster of paricles in yz-plane

bull Section 33 provides the motion of the particles while following a line tra-jectory

bull Section 34 provides the motion of the particles while following a sinwave

3 Chapter 4 concludes and provides directions for future work

bull Section 41 provides the conclusions of this work

bull Section 42 discusses suggestions pertaining to our future work

9

1 Introduction

4 Chapter 5 presnsents the appendix and references used for this thesis

10

2

Experiments preparation

21 Modeling of a KUKA IIWA Serial-Link Manipulator

KUKA iiwa R800 (7 DOFs robot KUKA GmbH Germany) was used in this workand an interface was implemented to control the robot

211 Robots in Minimally Invasive Interventions

The range of applications for medical robots has increased in recent years Hospitalsreport hundreds and even thousands of procedures performed with the da Vinci robotshown in Fig21 which now has the delicate ability to precisely peel a grape It isused in all kinds of surgeries from heart-valve repair to tumor removal and acrossvarious surgical disciplines The ROBODOC shown in Fig22 now combines withORTHODOC to preplan surgical procedures and carry them out with relatively highaccuracy ROBODOC also assists the physicians with knee replacements Roboticprosthetics now include sensors that are attached to the muscles and nerves allowingpatients to experience a sense of touch and to control movements with thought Pros-thetic hands allow fine motor skills such as writing typing or playing piano Robotsmight have feelings after all with the help of a robotic exoskeleton stroke victimscan regain lost arm movement quicker than with traditional physical therapy Able to

11

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

1

Introduction

11 Motivation

Cancer is considered as the world second leading cause of death after heart disease [1]Survival rates are increasing owing to the improved medical treatment However thecurrent methods of treatment have negative side-effects on the body such as painloss of balance malnutrition and blood disorder Therefore much effort has beenexpanded to achieve targeted therapy that can treat cancer more safely than conven-tional methods [2] This work investigates the utlilzation of magnetic drug carriers(paramagnetic microparticles) that can reach the targeted position via a magnetic-based robotic system This study is necessary to translate the microparticles intoin vivo application

12 Electromagnetic Systems and Magnetic Control

Magnetically-driven and self-driven medical microrobots have the potential to deliverdrugs to deep seated regions in the human body Paramagnetic microparticles andhelical robots are examples of magnetically-driven microrobots while magnetotacticbacteria is a type of self-driven microrobots that have the potential to controllably

5

1 Introduction

Figure 11 The OctoMag prototype is designed and constructed at ETH Zurich Thesystem contains eight electromagnets in closed configuration [6]

achieve targeted drug delivery [3] A study was made for magnetic nanoparticles withsurface bound drug molecules type of microrobots were injected into the vascular sys-tem upstream from malignant tissue and were captured at the tumor using a localapplied magnetic field [4] Iron-oxide nanoparticles have been used in vitro by achiev-ing selective penetration of MCF-7 breast cancer cell without causing any damage tothe memberane and without any effect on the cell morphology [5] These differenttypes of microrobots are controlled wirelessly using controlled magnetic fields whichcan be generated using several methods The first method uses electromagnets thatare simultaneously position and current controlled The second method uses position-controlled permanent magnets Closed-configurations systems are generally thoughtto be essential for diverse biomedical applications during in vitro experimentation [7][8] The OctoMag shown in Fig 11 is a type of closed-loop control magnetic systemwith computer vision implemented for accurate positioning [2] [6] The OctoMag isdesigned for controlling the intraocular microrobots in minimally invasive retinal ther-apy and diagnosis It is also has been used as a wireless micromanipulation systemunder a light microscope Due to several technological barriers these systems cannotbe scaled up to the size of invivo applications [9] as it is generally not viable to achieverelatively high gradients at a distance They are placed in a way to form a closed-

6

12 Electromagnetic Systems and Magnetic Control

Figure 12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltechnolo-gie GmbH Rostock-Warnemuende Germany) in three-dimensional spaceThe magnetic system consists of 8 iron-core electromagnets that surroundsa water reservoir Magnetic-based robotic system give more freedom inmoving through trajectories compared to this system However this sys-tem showed greater positioning accuracy compared to the magnet-basedone [36]

configuration where the workspace containing microrobot which can be in the formof paramagnetic microparticles and nanoparticles The advantage of these particlesover the other types of microrobots is that they are biodegradable which give themthe opportunity to be used in most clinical applications However the magnetic forcesextended on these particles are small due to the dependency of the magnetic force onthe size and geometry of the object itself This problem can be solved by increasingthe directional derivatives in the applied field found between these coils (usually 4 or 8electromagentic coils) as shown Fig 12 The limited projection distance of the mag-netic field gradient constrains the workspace of these systems to tens of millimeters atmost

It is also difficult to integrate a clinical imaging modality to magnetic systems with

7

1 Introduction

closed configuration [10] [13] for instance These challenges have been partially over-come by Mahoney et al [9] [11] A combination of magnetic field-driven helicalmicrorobots [12] [15] and a robotic arm with a fixed permanent magnet to its endef-fector has been proposed and used to achieve locomotion throughout relatively largeworkspace in 3D space The rotating dipole field enables the microrobot to achievehelical propulsion while the robotic arm allows the rotating permanent magnet togenerate the rotating fields at any point within its workspace Recently Nelson etal [16] have utilized the non-uniformity of the rotating field in generating two inde-pendent rotating magnetic fields using a single magnet dipole An open configurationof two synchronized rotating dipole fields has been also introduced by Alshafeei etal to control the motion of helical microrobot inside a catheter segment [17] and in3D space [18] The open configuration of this magnetic-based robotic system enablesscaling to the size of in vivo applications and its ability to remove blood clots hasbeen demonstrated in vitro [19] An electromagnetic coil and a permanent magnethave been integrated to the end-effector of a robotic arm with four degrees-of-freedom(DOFs) in [20] and a comparative study has shown the ability of the coil to achievehigher positioning accuracy of paramagnetic microparticles compared to the perma-nent magnet Although electromagnetic coils provide greater positioning accuracyowing to the ability to control the magnitude of the field gradient via the currentinput it is desirable to benefit from the higher magnetic field and field gradient thatcan be generated using permanent magnets A theoretical analysis and experimen-tal tele-manipulation of trapped paramagnetic micro-particles inside gas bubbles wasintroduced by our research group Khalil et al by analyzing the interaction forces ofa trapped micro-particle in a gas bubble at the water-gas interface and estimatingthe interaction forces between the micro-particles and the bubble using a calibratedforce observer [22] Also another study was conducted for expermintly investigationthe non-contact manipulation of the microbeads using a cluster of microparticles Thenon-contact micromanipulation achieves successful releases by moving the microbeadstowards the reference positions [23]In this work we design a closed-loop control system to position clusters of param-agnetic microparticles in low-Reynolds number fluids The control is achieved in 3Dspace using a robotic arm with seven DOFs and a permanent magnet First we derivea mapping between the position of the cluster of paramagnetic microparticles and theconfiguration of the robotic arm This mapping is used as basis of a closed-loop controlsystem to position the cluster in 3D space

8

13 Thesis Structure

13 Thesis Structure

This thesis presents a control frame work to achieve positioning of drug carriers (para-magnetic magneticparticles) in 3D space using a magent-based robot in a closed-loopconfiguration The thesis is structured as follows

1 Chapter 2 describes the motion control and modeling of the paramagnetic mi-croparticles in 3D space

bull Section 21 presents the interface of the robotic arm (KUKA iiwa) thecommunication protocol used and the two sides of the interface in java andC

bull Section 22 explains the camera calibration to extract the internsic param-eters of the two cameras used in the experiments

bull Section 23 provides the image processing used to track the particle in real-time

bull Section 24 provides the kinematic model of the robotic arm and the mi-croparticles and the closed-loop control system design

2 Chapter 3 presents the experiments part showing the different positioning ofthe particles in the 3D space

bull Section 31 provides the control of a cluster of particles in xy-plane

bull Section 32 provides the control of a cluster of paricles in yz-plane

bull Section 33 provides the motion of the particles while following a line tra-jectory

bull Section 34 provides the motion of the particles while following a sinwave

3 Chapter 4 concludes and provides directions for future work

bull Section 41 provides the conclusions of this work

bull Section 42 discusses suggestions pertaining to our future work

9

1 Introduction

4 Chapter 5 presnsents the appendix and references used for this thesis

10

2

Experiments preparation

21 Modeling of a KUKA IIWA Serial-Link Manipulator

KUKA iiwa R800 (7 DOFs robot KUKA GmbH Germany) was used in this workand an interface was implemented to control the robot

211 Robots in Minimally Invasive Interventions

The range of applications for medical robots has increased in recent years Hospitalsreport hundreds and even thousands of procedures performed with the da Vinci robotshown in Fig21 which now has the delicate ability to precisely peel a grape It isused in all kinds of surgeries from heart-valve repair to tumor removal and acrossvarious surgical disciplines The ROBODOC shown in Fig22 now combines withORTHODOC to preplan surgical procedures and carry them out with relatively highaccuracy ROBODOC also assists the physicians with knee replacements Roboticprosthetics now include sensors that are attached to the muscles and nerves allowingpatients to experience a sense of touch and to control movements with thought Pros-thetic hands allow fine motor skills such as writing typing or playing piano Robotsmight have feelings after all with the help of a robotic exoskeleton stroke victimscan regain lost arm movement quicker than with traditional physical therapy Able to

11

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

1 Introduction

Figure 11 The OctoMag prototype is designed and constructed at ETH Zurich Thesystem contains eight electromagnets in closed configuration [6]

achieve targeted drug delivery [3] A study was made for magnetic nanoparticles withsurface bound drug molecules type of microrobots were injected into the vascular sys-tem upstream from malignant tissue and were captured at the tumor using a localapplied magnetic field [4] Iron-oxide nanoparticles have been used in vitro by achiev-ing selective penetration of MCF-7 breast cancer cell without causing any damage tothe memberane and without any effect on the cell morphology [5] These differenttypes of microrobots are controlled wirelessly using controlled magnetic fields whichcan be generated using several methods The first method uses electromagnets thatare simultaneously position and current controlled The second method uses position-controlled permanent magnets Closed-configurations systems are generally thoughtto be essential for diverse biomedical applications during in vitro experimentation [7][8] The OctoMag shown in Fig 11 is a type of closed-loop control magnetic systemwith computer vision implemented for accurate positioning [2] [6] The OctoMag isdesigned for controlling the intraocular microrobots in minimally invasive retinal ther-apy and diagnosis It is also has been used as a wireless micromanipulation systemunder a light microscope Due to several technological barriers these systems cannotbe scaled up to the size of invivo applications [9] as it is generally not viable to achieverelatively high gradients at a distance They are placed in a way to form a closed-

6

12 Electromagnetic Systems and Magnetic Control

Figure 12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltechnolo-gie GmbH Rostock-Warnemuende Germany) in three-dimensional spaceThe magnetic system consists of 8 iron-core electromagnets that surroundsa water reservoir Magnetic-based robotic system give more freedom inmoving through trajectories compared to this system However this sys-tem showed greater positioning accuracy compared to the magnet-basedone [36]

configuration where the workspace containing microrobot which can be in the formof paramagnetic microparticles and nanoparticles The advantage of these particlesover the other types of microrobots is that they are biodegradable which give themthe opportunity to be used in most clinical applications However the magnetic forcesextended on these particles are small due to the dependency of the magnetic force onthe size and geometry of the object itself This problem can be solved by increasingthe directional derivatives in the applied field found between these coils (usually 4 or 8electromagentic coils) as shown Fig 12 The limited projection distance of the mag-netic field gradient constrains the workspace of these systems to tens of millimeters atmost

It is also difficult to integrate a clinical imaging modality to magnetic systems with

7

1 Introduction

closed configuration [10] [13] for instance These challenges have been partially over-come by Mahoney et al [9] [11] A combination of magnetic field-driven helicalmicrorobots [12] [15] and a robotic arm with a fixed permanent magnet to its endef-fector has been proposed and used to achieve locomotion throughout relatively largeworkspace in 3D space The rotating dipole field enables the microrobot to achievehelical propulsion while the robotic arm allows the rotating permanent magnet togenerate the rotating fields at any point within its workspace Recently Nelson etal [16] have utilized the non-uniformity of the rotating field in generating two inde-pendent rotating magnetic fields using a single magnet dipole An open configurationof two synchronized rotating dipole fields has been also introduced by Alshafeei etal to control the motion of helical microrobot inside a catheter segment [17] and in3D space [18] The open configuration of this magnetic-based robotic system enablesscaling to the size of in vivo applications and its ability to remove blood clots hasbeen demonstrated in vitro [19] An electromagnetic coil and a permanent magnethave been integrated to the end-effector of a robotic arm with four degrees-of-freedom(DOFs) in [20] and a comparative study has shown the ability of the coil to achievehigher positioning accuracy of paramagnetic microparticles compared to the perma-nent magnet Although electromagnetic coils provide greater positioning accuracyowing to the ability to control the magnitude of the field gradient via the currentinput it is desirable to benefit from the higher magnetic field and field gradient thatcan be generated using permanent magnets A theoretical analysis and experimen-tal tele-manipulation of trapped paramagnetic micro-particles inside gas bubbles wasintroduced by our research group Khalil et al by analyzing the interaction forces ofa trapped micro-particle in a gas bubble at the water-gas interface and estimatingthe interaction forces between the micro-particles and the bubble using a calibratedforce observer [22] Also another study was conducted for expermintly investigationthe non-contact manipulation of the microbeads using a cluster of microparticles Thenon-contact micromanipulation achieves successful releases by moving the microbeadstowards the reference positions [23]In this work we design a closed-loop control system to position clusters of param-agnetic microparticles in low-Reynolds number fluids The control is achieved in 3Dspace using a robotic arm with seven DOFs and a permanent magnet First we derivea mapping between the position of the cluster of paramagnetic microparticles and theconfiguration of the robotic arm This mapping is used as basis of a closed-loop controlsystem to position the cluster in 3D space

8

13 Thesis Structure

13 Thesis Structure

This thesis presents a control frame work to achieve positioning of drug carriers (para-magnetic magneticparticles) in 3D space using a magent-based robot in a closed-loopconfiguration The thesis is structured as follows

1 Chapter 2 describes the motion control and modeling of the paramagnetic mi-croparticles in 3D space

bull Section 21 presents the interface of the robotic arm (KUKA iiwa) thecommunication protocol used and the two sides of the interface in java andC

bull Section 22 explains the camera calibration to extract the internsic param-eters of the two cameras used in the experiments

bull Section 23 provides the image processing used to track the particle in real-time

bull Section 24 provides the kinematic model of the robotic arm and the mi-croparticles and the closed-loop control system design

2 Chapter 3 presents the experiments part showing the different positioning ofthe particles in the 3D space

bull Section 31 provides the control of a cluster of particles in xy-plane

bull Section 32 provides the control of a cluster of paricles in yz-plane

bull Section 33 provides the motion of the particles while following a line tra-jectory

bull Section 34 provides the motion of the particles while following a sinwave

3 Chapter 4 concludes and provides directions for future work

bull Section 41 provides the conclusions of this work

bull Section 42 discusses suggestions pertaining to our future work

9

1 Introduction

4 Chapter 5 presnsents the appendix and references used for this thesis

10

2

Experiments preparation

21 Modeling of a KUKA IIWA Serial-Link Manipulator

KUKA iiwa R800 (7 DOFs robot KUKA GmbH Germany) was used in this workand an interface was implemented to control the robot

211 Robots in Minimally Invasive Interventions

The range of applications for medical robots has increased in recent years Hospitalsreport hundreds and even thousands of procedures performed with the da Vinci robotshown in Fig21 which now has the delicate ability to precisely peel a grape It isused in all kinds of surgeries from heart-valve repair to tumor removal and acrossvarious surgical disciplines The ROBODOC shown in Fig22 now combines withORTHODOC to preplan surgical procedures and carry them out with relatively highaccuracy ROBODOC also assists the physicians with knee replacements Roboticprosthetics now include sensors that are attached to the muscles and nerves allowingpatients to experience a sense of touch and to control movements with thought Pros-thetic hands allow fine motor skills such as writing typing or playing piano Robotsmight have feelings after all with the help of a robotic exoskeleton stroke victimscan regain lost arm movement quicker than with traditional physical therapy Able to

11

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

12 Electromagnetic Systems and Magnetic Control

Figure 12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltechnolo-gie GmbH Rostock-Warnemuende Germany) in three-dimensional spaceThe magnetic system consists of 8 iron-core electromagnets that surroundsa water reservoir Magnetic-based robotic system give more freedom inmoving through trajectories compared to this system However this sys-tem showed greater positioning accuracy compared to the magnet-basedone [36]

configuration where the workspace containing microrobot which can be in the formof paramagnetic microparticles and nanoparticles The advantage of these particlesover the other types of microrobots is that they are biodegradable which give themthe opportunity to be used in most clinical applications However the magnetic forcesextended on these particles are small due to the dependency of the magnetic force onthe size and geometry of the object itself This problem can be solved by increasingthe directional derivatives in the applied field found between these coils (usually 4 or 8electromagentic coils) as shown Fig 12 The limited projection distance of the mag-netic field gradient constrains the workspace of these systems to tens of millimeters atmost

It is also difficult to integrate a clinical imaging modality to magnetic systems with

7

1 Introduction

closed configuration [10] [13] for instance These challenges have been partially over-come by Mahoney et al [9] [11] A combination of magnetic field-driven helicalmicrorobots [12] [15] and a robotic arm with a fixed permanent magnet to its endef-fector has been proposed and used to achieve locomotion throughout relatively largeworkspace in 3D space The rotating dipole field enables the microrobot to achievehelical propulsion while the robotic arm allows the rotating permanent magnet togenerate the rotating fields at any point within its workspace Recently Nelson etal [16] have utilized the non-uniformity of the rotating field in generating two inde-pendent rotating magnetic fields using a single magnet dipole An open configurationof two synchronized rotating dipole fields has been also introduced by Alshafeei etal to control the motion of helical microrobot inside a catheter segment [17] and in3D space [18] The open configuration of this magnetic-based robotic system enablesscaling to the size of in vivo applications and its ability to remove blood clots hasbeen demonstrated in vitro [19] An electromagnetic coil and a permanent magnethave been integrated to the end-effector of a robotic arm with four degrees-of-freedom(DOFs) in [20] and a comparative study has shown the ability of the coil to achievehigher positioning accuracy of paramagnetic microparticles compared to the perma-nent magnet Although electromagnetic coils provide greater positioning accuracyowing to the ability to control the magnitude of the field gradient via the currentinput it is desirable to benefit from the higher magnetic field and field gradient thatcan be generated using permanent magnets A theoretical analysis and experimen-tal tele-manipulation of trapped paramagnetic micro-particles inside gas bubbles wasintroduced by our research group Khalil et al by analyzing the interaction forces ofa trapped micro-particle in a gas bubble at the water-gas interface and estimatingthe interaction forces between the micro-particles and the bubble using a calibratedforce observer [22] Also another study was conducted for expermintly investigationthe non-contact manipulation of the microbeads using a cluster of microparticles Thenon-contact micromanipulation achieves successful releases by moving the microbeadstowards the reference positions [23]In this work we design a closed-loop control system to position clusters of param-agnetic microparticles in low-Reynolds number fluids The control is achieved in 3Dspace using a robotic arm with seven DOFs and a permanent magnet First we derivea mapping between the position of the cluster of paramagnetic microparticles and theconfiguration of the robotic arm This mapping is used as basis of a closed-loop controlsystem to position the cluster in 3D space

8

13 Thesis Structure

13 Thesis Structure

This thesis presents a control frame work to achieve positioning of drug carriers (para-magnetic magneticparticles) in 3D space using a magent-based robot in a closed-loopconfiguration The thesis is structured as follows

1 Chapter 2 describes the motion control and modeling of the paramagnetic mi-croparticles in 3D space

bull Section 21 presents the interface of the robotic arm (KUKA iiwa) thecommunication protocol used and the two sides of the interface in java andC

bull Section 22 explains the camera calibration to extract the internsic param-eters of the two cameras used in the experiments

bull Section 23 provides the image processing used to track the particle in real-time

bull Section 24 provides the kinematic model of the robotic arm and the mi-croparticles and the closed-loop control system design

2 Chapter 3 presents the experiments part showing the different positioning ofthe particles in the 3D space

bull Section 31 provides the control of a cluster of particles in xy-plane

bull Section 32 provides the control of a cluster of paricles in yz-plane

bull Section 33 provides the motion of the particles while following a line tra-jectory

bull Section 34 provides the motion of the particles while following a sinwave

3 Chapter 4 concludes and provides directions for future work

bull Section 41 provides the conclusions of this work

bull Section 42 discusses suggestions pertaining to our future work

9

1 Introduction

4 Chapter 5 presnsents the appendix and references used for this thesis

10

2

Experiments preparation

21 Modeling of a KUKA IIWA Serial-Link Manipulator

KUKA iiwa R800 (7 DOFs robot KUKA GmbH Germany) was used in this workand an interface was implemented to control the robot

211 Robots in Minimally Invasive Interventions

The range of applications for medical robots has increased in recent years Hospitalsreport hundreds and even thousands of procedures performed with the da Vinci robotshown in Fig21 which now has the delicate ability to precisely peel a grape It isused in all kinds of surgeries from heart-valve repair to tumor removal and acrossvarious surgical disciplines The ROBODOC shown in Fig22 now combines withORTHODOC to preplan surgical procedures and carry them out with relatively highaccuracy ROBODOC also assists the physicians with knee replacements Roboticprosthetics now include sensors that are attached to the muscles and nerves allowingpatients to experience a sense of touch and to control movements with thought Pros-thetic hands allow fine motor skills such as writing typing or playing piano Robotsmight have feelings after all with the help of a robotic exoskeleton stroke victimscan regain lost arm movement quicker than with traditional physical therapy Able to

11

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

1 Introduction

closed configuration [10] [13] for instance These challenges have been partially over-come by Mahoney et al [9] [11] A combination of magnetic field-driven helicalmicrorobots [12] [15] and a robotic arm with a fixed permanent magnet to its endef-fector has been proposed and used to achieve locomotion throughout relatively largeworkspace in 3D space The rotating dipole field enables the microrobot to achievehelical propulsion while the robotic arm allows the rotating permanent magnet togenerate the rotating fields at any point within its workspace Recently Nelson etal [16] have utilized the non-uniformity of the rotating field in generating two inde-pendent rotating magnetic fields using a single magnet dipole An open configurationof two synchronized rotating dipole fields has been also introduced by Alshafeei etal to control the motion of helical microrobot inside a catheter segment [17] and in3D space [18] The open configuration of this magnetic-based robotic system enablesscaling to the size of in vivo applications and its ability to remove blood clots hasbeen demonstrated in vitro [19] An electromagnetic coil and a permanent magnethave been integrated to the end-effector of a robotic arm with four degrees-of-freedom(DOFs) in [20] and a comparative study has shown the ability of the coil to achievehigher positioning accuracy of paramagnetic microparticles compared to the perma-nent magnet Although electromagnetic coils provide greater positioning accuracyowing to the ability to control the magnitude of the field gradient via the currentinput it is desirable to benefit from the higher magnetic field and field gradient thatcan be generated using permanent magnets A theoretical analysis and experimen-tal tele-manipulation of trapped paramagnetic micro-particles inside gas bubbles wasintroduced by our research group Khalil et al by analyzing the interaction forces ofa trapped micro-particle in a gas bubble at the water-gas interface and estimatingthe interaction forces between the micro-particles and the bubble using a calibratedforce observer [22] Also another study was conducted for expermintly investigationthe non-contact manipulation of the microbeads using a cluster of microparticles Thenon-contact micromanipulation achieves successful releases by moving the microbeadstowards the reference positions [23]In this work we design a closed-loop control system to position clusters of param-agnetic microparticles in low-Reynolds number fluids The control is achieved in 3Dspace using a robotic arm with seven DOFs and a permanent magnet First we derivea mapping between the position of the cluster of paramagnetic microparticles and theconfiguration of the robotic arm This mapping is used as basis of a closed-loop controlsystem to position the cluster in 3D space

8

13 Thesis Structure

13 Thesis Structure

This thesis presents a control frame work to achieve positioning of drug carriers (para-magnetic magneticparticles) in 3D space using a magent-based robot in a closed-loopconfiguration The thesis is structured as follows

1 Chapter 2 describes the motion control and modeling of the paramagnetic mi-croparticles in 3D space

bull Section 21 presents the interface of the robotic arm (KUKA iiwa) thecommunication protocol used and the two sides of the interface in java andC

bull Section 22 explains the camera calibration to extract the internsic param-eters of the two cameras used in the experiments

bull Section 23 provides the image processing used to track the particle in real-time

bull Section 24 provides the kinematic model of the robotic arm and the mi-croparticles and the closed-loop control system design

2 Chapter 3 presents the experiments part showing the different positioning ofthe particles in the 3D space

bull Section 31 provides the control of a cluster of particles in xy-plane

bull Section 32 provides the control of a cluster of paricles in yz-plane

bull Section 33 provides the motion of the particles while following a line tra-jectory

bull Section 34 provides the motion of the particles while following a sinwave

3 Chapter 4 concludes and provides directions for future work

bull Section 41 provides the conclusions of this work

bull Section 42 discusses suggestions pertaining to our future work

9

1 Introduction

4 Chapter 5 presnsents the appendix and references used for this thesis

10

2

Experiments preparation

21 Modeling of a KUKA IIWA Serial-Link Manipulator

KUKA iiwa R800 (7 DOFs robot KUKA GmbH Germany) was used in this workand an interface was implemented to control the robot

211 Robots in Minimally Invasive Interventions

The range of applications for medical robots has increased in recent years Hospitalsreport hundreds and even thousands of procedures performed with the da Vinci robotshown in Fig21 which now has the delicate ability to precisely peel a grape It isused in all kinds of surgeries from heart-valve repair to tumor removal and acrossvarious surgical disciplines The ROBODOC shown in Fig22 now combines withORTHODOC to preplan surgical procedures and carry them out with relatively highaccuracy ROBODOC also assists the physicians with knee replacements Roboticprosthetics now include sensors that are attached to the muscles and nerves allowingpatients to experience a sense of touch and to control movements with thought Pros-thetic hands allow fine motor skills such as writing typing or playing piano Robotsmight have feelings after all with the help of a robotic exoskeleton stroke victimscan regain lost arm movement quicker than with traditional physical therapy Able to

11

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

13 Thesis Structure

13 Thesis Structure

This thesis presents a control frame work to achieve positioning of drug carriers (para-magnetic magneticparticles) in 3D space using a magent-based robot in a closed-loopconfiguration The thesis is structured as follows

1 Chapter 2 describes the motion control and modeling of the paramagnetic mi-croparticles in 3D space

bull Section 21 presents the interface of the robotic arm (KUKA iiwa) thecommunication protocol used and the two sides of the interface in java andC

bull Section 22 explains the camera calibration to extract the internsic param-eters of the two cameras used in the experiments

bull Section 23 provides the image processing used to track the particle in real-time

bull Section 24 provides the kinematic model of the robotic arm and the mi-croparticles and the closed-loop control system design

2 Chapter 3 presents the experiments part showing the different positioning ofthe particles in the 3D space

bull Section 31 provides the control of a cluster of particles in xy-plane

bull Section 32 provides the control of a cluster of paricles in yz-plane

bull Section 33 provides the motion of the particles while following a line tra-jectory

bull Section 34 provides the motion of the particles while following a sinwave

3 Chapter 4 concludes and provides directions for future work

bull Section 41 provides the conclusions of this work

bull Section 42 discusses suggestions pertaining to our future work

9

1 Introduction

4 Chapter 5 presnsents the appendix and references used for this thesis

10

2

Experiments preparation

21 Modeling of a KUKA IIWA Serial-Link Manipulator

KUKA iiwa R800 (7 DOFs robot KUKA GmbH Germany) was used in this workand an interface was implemented to control the robot

211 Robots in Minimally Invasive Interventions

The range of applications for medical robots has increased in recent years Hospitalsreport hundreds and even thousands of procedures performed with the da Vinci robotshown in Fig21 which now has the delicate ability to precisely peel a grape It isused in all kinds of surgeries from heart-valve repair to tumor removal and acrossvarious surgical disciplines The ROBODOC shown in Fig22 now combines withORTHODOC to preplan surgical procedures and carry them out with relatively highaccuracy ROBODOC also assists the physicians with knee replacements Roboticprosthetics now include sensors that are attached to the muscles and nerves allowingpatients to experience a sense of touch and to control movements with thought Pros-thetic hands allow fine motor skills such as writing typing or playing piano Robotsmight have feelings after all with the help of a robotic exoskeleton stroke victimscan regain lost arm movement quicker than with traditional physical therapy Able to

11

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

1 Introduction

4 Chapter 5 presnsents the appendix and references used for this thesis

10

2

Experiments preparation

21 Modeling of a KUKA IIWA Serial-Link Manipulator

KUKA iiwa R800 (7 DOFs robot KUKA GmbH Germany) was used in this workand an interface was implemented to control the robot

211 Robots in Minimally Invasive Interventions

The range of applications for medical robots has increased in recent years Hospitalsreport hundreds and even thousands of procedures performed with the da Vinci robotshown in Fig21 which now has the delicate ability to precisely peel a grape It isused in all kinds of surgeries from heart-valve repair to tumor removal and acrossvarious surgical disciplines The ROBODOC shown in Fig22 now combines withORTHODOC to preplan surgical procedures and carry them out with relatively highaccuracy ROBODOC also assists the physicians with knee replacements Roboticprosthetics now include sensors that are attached to the muscles and nerves allowingpatients to experience a sense of touch and to control movements with thought Pros-thetic hands allow fine motor skills such as writing typing or playing piano Robotsmight have feelings after all with the help of a robotic exoskeleton stroke victimscan regain lost arm movement quicker than with traditional physical therapy Able to

11

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

2

Experiments preparation

21 Modeling of a KUKA IIWA Serial-Link Manipulator

KUKA iiwa R800 (7 DOFs robot KUKA GmbH Germany) was used in this workand an interface was implemented to control the robot

211 Robots in Minimally Invasive Interventions

The range of applications for medical robots has increased in recent years Hospitalsreport hundreds and even thousands of procedures performed with the da Vinci robotshown in Fig21 which now has the delicate ability to precisely peel a grape It isused in all kinds of surgeries from heart-valve repair to tumor removal and acrossvarious surgical disciplines The ROBODOC shown in Fig22 now combines withORTHODOC to preplan surgical procedures and carry them out with relatively highaccuracy ROBODOC also assists the physicians with knee replacements Roboticprosthetics now include sensors that are attached to the muscles and nerves allowingpatients to experience a sense of touch and to control movements with thought Pros-thetic hands allow fine motor skills such as writing typing or playing piano Robotsmight have feelings after all with the help of a robotic exoskeleton stroke victimscan regain lost arm movement quicker than with traditional physical therapy Able to

11

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

2 Experiments preparation

mimic 95 of fully functional human arms patients strap on the robotic arms andperform a series of exercises Sensors detect muscle strength range of motion andalso brain activity informing therapists of progress The machine also helps retrainthe brain enabling healthy areas to compensate for stroke-damaged ones Roboticsystems also help patients relearn walking and other motor skills

Robotic surgery has accomplished its goal of removing the limitations of laparoscopicsurgery It has returned three-dimensional visualization to internal surgical spaces andeven slightly increased the normal range of motion Very fine movements are possibleas evidenced by the grape-peeling demonstration furthermore the ability to perform360-degree rotations with delicate instruments further increases the capabilities ofsurgeons The other laparoscopic advantages of reduced blood loss less pain loweredrisk of infection and quicker recovery time are all enhanced with the precision ofrobotic instruments Nowadays robot arms are more widely used in most of industriescompetition from companies are rising more Itrsquos always a tradeoff between price andprecision

212 KUKA iiwa r800

In this thesis we used KUKArsquos robot KUKA iiwa r800 as shown in Fig 23 A 7axis robot which gives the robot flexibility to move in the null space This robot issupporting our proposed idea of this thesis that in future robot arms can be used withthe doctors easily in operations like cancer fighting Moreover this robot can be learnsome movements then repeats them precisely

213 Interface of the KUKA robot

User friendly environment is the main thing one can think of when proposing an ideaSo in our case we wanted to implement a windows application that can be easily usedto control the robot without touching the robots panel Used C as the programminglanguage to create the application The purpose of the application is to receive thecurrent positions forces time-stamp inputsoutputs and to be able to send therobot commands to go to certain position change the stiffness of specific axis changethe damping in specific direction or axis and finally able to make the robot go intogravity compensation mode where the user can hold the gripper and move the robotin any direction easily

12

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console where thesurgeon sits while operating a patient-side cart where the patient is posi-tioned during surgery interactive robotic arms a three-dimensional highdefinition vision system and proprietary EndoWrist instruments [28] (a)The EndoWrist Insturment (b) Surgen console (c) vision system (d)Patient-side cart

13

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

2 Experiments preparation

Figure 22 ROBODOC Surgical system that enables pre-surgical 3D planning Mean-while the ROBODOC Surgical assistant executes the pre-surgical planwith unparalleled precision Benefits of the ROBODOC System over tradi-tional methods include improved cavity fit fill and alignment for prostheticimplants [29] [30] [32] as well as a reduction of leg length discrepancy [30]and intra-operative fractures [29] [31]

2131 Communication protocols for connecting the robot server with a capplication

A communications protocol defines the rules for sending blocks of data (each known asa Protocol Data Unit (PDU)) from one node in a network to another node Protocolsare normally defined in a layered manner and provide all or part of the services spec-ified by a layer 24 Two famous examples of the widely used protocols are TCPIP(Transmission Control ProtocolInternet Protocol) and UDP (User Datagram Proto-col) although both are used to handle data communications between terminals in anIP network and both are located in Transport Layer referring to Figure 24 howeverthere are major differences between these two protocols TCP is connection orientedprotocol and once a connection is established data can be sent bidirectionally onthe other hand UDP is a simpler connectionless internet protocol Multiple messagesare sent as packets in chunks using UDP Also the speed of transfer in TCP is slowerthan UDP However TCP is more reliable as it guarantees that the data transferred

14

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

21 Modeling of a KUKA IIWA Serial-Link Manipulator

Figure 23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom

remains intact and arrives in the same order in which it was sent TCP send acknowl-edgment segments when data is received wheras UDP protocol just ignores sendignacknowloedgments and assumes data is recieved by the other end [41]

When it comes to choosing between these two protocols TCP is suited for applicationsthat requires high reliability and transmission time is relatively less critical while UDPis suitable for applications that need fast efficient transmission UDP is also usefulfor servers that answer small queries from huge numbers of clients

214 Additional features added to the robot arm to allow a free motionin the null frame and motion to a target position

Gravity compensation and motion of the robot towards a reference position wereimplemented in the interface of the robot More features could be added in the futurefor example teaching the robot a path

15

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

2 Experiments preparation

Figure 24 Communication protocol architecture consisting of 5 layers It starts withthe physical layer that deals with the physical (ie electrical and mechani-cal) aspects of transmitting data (eg voltage levels pin-connector designcable lengths and grounding arrangements) the Data-link layer deals withthe transmission of data frames (eg packets) over a physical link betweennetwork entities including the incorporation of error-correction coding intothe data frames the network that deals with establishing paths for databetween a pair of computers and handling any switching among alterna-tive routes between the computers as well as with definitions of how tobreak messages (or packets) up into individual packets of data in such away that the packets can be transmitted and then reassembled Transportlayer deals with data transfer between end systems flow control for twocomputers Finally application layer that deals with the interface betweena user and the host computer

21401 Gravity compensation feature of allowing robot arm motion in the nullspaceA gravity compensation mode is programmed for this robot a button attached to ahandler as shown in Fig 25 to one of the inputs of the robot Button is pressed thecartesian stiffness of the robot are set to zero in all positions and orientations andwhen release the stiffness are set to the maximum in X Y and Z to 5000 Nm whilein A B and C to 200 Nm

21402 Motion feature for driving the robot arm to an aimed positionA Computer Aided Design (CAD) file can be imported into our software and byselecting one triangle A cartesian aimed position is automatically calculated Thisaimed position show the right orientation of the camera and distance from the cad

16

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

21 Modeling of a KUKA IIWA Serial-Link Manipulator

GravityCompButton

Figure 25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero

file this feature is created for the camera to take a good sharp image but it can bealso used in medical application to make sure the robot end effector that holds themagnet is in the right distance and orientation from the patient to assure right bestmagnetic fields to drive the magnetic drugs to the right position

The calculation of the Aimed position is as follow

1 Select a triangle on the CAD file

2 Calculate the center of the selected triangle

xc = x1 + x2 + x33 (21)

where the center of the triangle in the xminusaxis is calculated using selected trianglevertices x1 is the the selected triangle vertices 1 in x x2 is the the selected trianglevertices 2 in x and x3 is the the selected triangle vertices 3 in x

17

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

2 Experiments preparation

yc = y1 + y2 + y33 (22)

where the center of the triangle in the yminusaxis is calculated using selected trianglevertices y1 is the the selected triangle vertices 1 in y y2 is the the selected trianglevertices 2 in y and y3 is the the selected triangle vertices 3 in y

zc = z1 + z2 + z33 (23)

where the center of the triangle in the zminusaxis is calculated using selected trianglevertices z1 is the the selected triangle vertices 1 in z z2 is the the selected trianglevertices 2 in z and z3 is the the selected triangle vertices 3 in z

TC =

TcxTcyTcz

(24)

where TC is the center of the triangle Tcx Tcy and Tcz represent the center of thetriangle in xminusaxis center of the triangle in yminusaxis and center of the triangle in zminusaxisrespectively and they are calculated in equations (21) (22) and (23)

3 put the vectors and position in a 4times4 transformation Matrix

T =

exx exy exz xceyx eyy eyz ycezx ezy ezz zc0 0 0 1

(25)

where exx exy exz eyx eyy eyz ezx ezy ezz represents the rotation matrix

4 Extracting the cartesian position translate postions are given by xc yc and zcrespectively while the angles are defined asThe yaw angle of the cartesian position (rad) is given by

a = tanminus1(eyxexx

) (26)

18

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

22 Camera calibration to extract the intrinsic parameters

Selected Tri

Figure 26 Aimed position on a selected triangle in a Computer Aided Device file

The pitch angle of the cartesian position (rad) is given by

b = tanminus1

ezxradice2zy + e2

zz

(27)

The roll angle of the cartesian position (rad) is given by

c = tanminus1(ezyezz

) (28)

An example of the software is shown in Fig 26 by selecting one triangle in theCAD file an aimed position is calculated and plotted in the software with the rightorientation and position from the triangle

22 Camera calibration to extract the intrinsic parameters

The intrinisic parameters of a camera fix the relation between image pixel coordinatesand ray directions in 3D camera coordinates system

19

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

2 Experiments preparation

221 Camera modelling

Models of the camera image are based on the model of the central perspective [39]in the so-called Lochkameramodell and the building-up advanced camera model Ta-ble 21 shows some terms with their meaning [40] [39]

Figure 27 Point calibration grid with orientation beam used to calibrate the camerasand extract the intrinsic parameters

The extended camera model describes the image of a point with 3D coordinates to a2D image area as shown in Fig 28 The point Pc is set to the point PI

In reality the projection center is in front of the image plane However the simplifi-cation in Fig28 that was made give advantage of a simple graphical representationThe optical image is divided into three steps The first step is the external trans-formation The second step is the perspective transformation The third step is oneinternal transformation starting with the extrnal transformation the projection cen-ter of the camera is the origin Z of the camera coordinate system This is located ina camera in the lens and therefore practically can not be measured Therefore thecamera coordinate system is set to a freely selectable world coordinate system In thisworld coordinate system the position and orientation from the camera frame with thetransformation wTc with homogeneous coordinates the relationship between point inthe world frame and point in the camera frame can be explained as

20

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

22 Camera calibration to extract the intrinsic parameters

Table 21 This table shows the definitions of the intrinsic parameters that areextracted

Projection center Z All points in the world coordinate sys-tem are defined in The projection cen-ter

Optical axis OA The straight line which is perpendic-ular to the image plane and Throughthe projection center becomes opticalAxis

Image coordinate system Image-Frame The image coordinate system is con-trolled by the axes And v The originis in the left Upper corner of the im-age The unit is pixels

Focal length F The Focal length fx or fy is the dis-tance Between the projection centerand the image plane The unit ispixels

Image starting point C The image main point is the intersec-tion of the optical Axis with the im-age plane and is imaged in the image-System with the parameters cx andcy

Camera coordinate system Camera-frame The camera coordinate system isthree-dimensional The origin beinglocated in the projection center zThe x and y axis are parallel to theaxes u and v respectively of the im-age coordinate system The z-axisPoints towards the scene The unitis millimeters

World coordinate system World-frame The world coordinate system is alsothree-dimensional And can be arbi-trary as a basic coordinate system Inthe room The unit is millimeters

The intrinsic parameter The intrinsic camera parameters areu0 v0 fx fy They are independentof the choice of the world coordinatesystem And remain at change Con-struction constant The unit is pixels

21

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

2 Experiments preparation

Figure 28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is the straightline which is perpendicular to the image plane and through the projec-tion center becomes optical axis CameraP point on the camera frameImageP point in the image frame wHc is the transformation from theworld to camera frame f focal length C is the image starting point andit is the interaction between the optical axis and the image plane

Pc = wTcPw = cTwPw (29)

The external parameters of the camera are included in the Matrix cTw That is whythis matrix also called extrinsic matrix The torque matrix has three degrees of free-dom which is the rotation angles about the three axes of the coordinate system Thetranslationhas three degrees of freedom This results in six freely selectable parametersfor the external Transformation

The second step is the prespective transformation The equation of the central projec-tion sets the perspective transformation of a point in the camera coordinate system Pcto the point Pp in the 2D sensor coordinate system fixed f describes the focal lengthThis is the distance of the image plane from the focal point

22

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

22 Camera calibration to extract the intrinsic parameters

Figure 29 This image show the yz-view and the xz-view of the camera In the yz-plane The CPy point is mapped to PPv using a beamset

The point cPy is mapped onto pPv and this leads to the following relationship

pPvf

=cPycPz

=gt pPv = fcPycPz

(210)

The relationship is applied on the same to the points cPx and pPu leading to

pPuf

=cPxcPz

=gt pPu = fcPxcPz

(211)

Substituting 210 and 211 we can extract the homogenous matrix as follows

cPz

pPucPzpPvcPz

= pTc

cPxcPycPz1

pTc =

f 0 0 00 f 0 00 0 1 0

(212)

A relation between cPz by s is given by

spPu

spPv

s

= pTccTw

wPxwPywPz

1

s = cPz (213)

23

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

2 Experiments preparation

Equation 213 allows 3D world coordinates to be transformed into 2D sensor coordi-nates This mapping process corresponds to a projective transformation of the Three-dimensional projective space in the two-dimensional projective plane The matrix pTc

is therefore also referred to as a perspective projection matrix

The third step is the internal transformation So far the unit of measurement of thecoordinate systems is still metric For the images to be converted into discrete imagecoordinates an internal transformation is necessary This transformation takes placeby a horizontal and vertical scale ku or kv Which translates into discrete image co-ordinates The discrete image matrix begins in the upper left corner of the imageTherefore a shift of the camera is main point and the optical axis with the imageplane into the camera main point This results in the displacement u0 in the x-planeof the camera and v0 in the y-plane These parameters can be combined in a lineartransformation matrix

ITIF =

ku 0 u00 kv v00 0 1

(214)

This equation can be expressed by

sIPu

sIPv

s

= ITppTccTw

wPxwPywPz

1

(215)

Setting fx=fku and fy=fkv yeilds

ITc = ITIF IFTc =

fx 0 u00 fy v00 0 1

(216)

The Matrix ITc is also referred to as intrinsic matrix because it is the intrinsic matrixParameters of a camera This allows the complete transformation of a 3D point Inworld coordinates on the camera coordinates into its image in the image coordinatesystem

sIPu

sIPv

s

= ITw

wPxwPywPz

1

(217)

24

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

22 Camera calibration to extract the intrinsic parameters

The Matrix cTw is the general projection matrix of the perspective projection

ITw = ITccTw =

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

(218)

There is now an unambiguous relationship between a point in World coordinate system3D point and its image on the 2D image plane in pixel coordinates manufactured Thiscan be explained by transforming Qij represent the components of the 3times4 projectionmatrix P

u = q11Xw + q12Yw + q13Zw + q14

q31Xw + q32Yw + q33Zw + q34 (219)

v = q21Xw + q22 Yw + q23Zw + q24q31Xw + q32 Yw + q33Zw + q34 (220)

It is only possible with knowledge of the projection matrix cTw by A scene depicted inthe image plane on geometric features of the imaged image object Before the cameracan be used as a sensor the ten camera parameters of the projection camera cTw arefirst determined with a calibration Six of them are the extrinsic parameters and theother four of the parameters are the intrinsic parameters six parameters can be thetransformation cTw of world coordinate system In camera coordinate system Theother four parameters Are the intrinsic parameters cx cy fx and fy with which arethe transformation ITc [39] This is done by recording 16 images of a calibration gridwith different orientations to the center of the camera An example of one calibrationgrid is shown in Figure 210

Using the two black bars can be a suitable image processing software coordinate systemin the image and then for all other points as a reference is used For a result ofsufficient quality this process must be carried out with an adequate accuracy Thecamera calibration is then the process in which the world coordinates And the pixelcoordinates of the m prominent points the ten unknown parameters to be appreciatedA simple approach for this is by means of a compensation calculation for exampleusing the least squares method to estimate the coefficients of the matrix P For thispurpose the equations 220 must be converted according to equation 221

[Xw Yw Zw 1 0 0 0 0 minusuXw minusuYw0 0 0 0 Xw Yw Zw 1 minusvXw minusvYw

] q11q34

=[uv

] (221)

25

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

2 Experiments preparation

Figure 210 The point calibration grid with orientation beam is implemented to ex-tract the intrinsic parameters of the camera [40]

Equation 221 can be assigned for each pair of pixel coordinates and world coordinatesAs there are ten unknowns at least five distinct points are necessary ten EquationsHowever this equation has three problems Firstly the world coordinates must notbe used in a plane because in this case Zw equals all equations and thus the columnsthree and four would be linearly dependent It would create a singular problem thatdoes not is released In order to circumvent this case a three-dimensional manufac-turing process would be required Secondly no lens distortion has yet been taken intoaccount which means that Model the reality only conditionally Thirdly the projec-tion matrix cTw it is not possible to make this unique in the extrinsic and intrinsicParameters For these three reasons equation 221 is for camera calibration As alter-natives there are a number of other methods These can basically be divided into twoCategories The first category uses non-linear optimization methods The unknownparameters This is done by an error function And this is subsequently minimized byvarying the unknown parameters This has many advantages Firstly this methodalso makes complicated camera- models because there is no limitation of the selectableparameters Another advantage is that a nonlinear optimization is very precise if theequations are the same reality and image converge Conversely disadvantages can also

26

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

23 Feature Tracking of Microparticles

be eliminated For the error function is wrongly formulated this causes instabilityor conceals there is a risk that the minimization will only hang in a local minimumalso The non-linear optimizations must be started with initial values When these aretoo Far away it can also happen that the error function is also in a local minimumfinishes An established method for extrinsic and intrinsic camera calibration is fromZsai Among other things it also takes into account the radial and tangential forcesdistortion The second method uses linear approaches not iteratively but directly fromthe equations the relationship between 3D points and their image on the image planedescribes a solution for the unknown Advantages of this processes are the stabilityand speed However such linear approaches are inaccurate A detailed overview ofexisting approaches [37] and [38]

23 Feature Tracking of Microparticles

An image processing algorithm is develeoped in order to detect the motion of themicroparticles inside the tube The first step for having right features of an image isachieved by calibrating cameras 22 to extract sharp images

We used digital image processing techniques to improve the Bartlane system then moreimprovements were added to these techniques in 1960s which improved significantllythe quality of the the moon images that was taken by the Ranger probe 7 Furthertill today more and more improvements are being added to these techniques and theuse of digital imaging processing spreaded widely between different fields for examplein Image enhancement Artistic effects Medical visualization Industrial inspectionLaw enfrocement etc

The steps of image processing can be explained as shown in figure 211

In our work we created a C program as shown in Fig 212 that take an image fromthe camera on a frame rate of 80 (Frames per Seconds) and process it until detectingthe particle The code used to track the paramagentic microparticles is shown in theappendix The steps that we did are as follow

bull Camera image is converted from RGB colored image to gray scale image so thatit is descriped with values between (0-255)

bull The gray scale image is then passed through threshold filter to be converted fromgray scale to a black white imageto delete all unnesessary items in the image

27

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

2 Experiments preparation

Figure 211 Digital image processing steps image acquisition is the first step or pro-cess of the fundamental steps of digital image processing Image acquisi-tion could be as simple as being given an image that is already in digitalform image enhancement is among the simplest and most appealing ar-eas of digital image processing The idea behind enhancement techniquesis to bring out detail that is obscured image restortion is an area thatalso deals with improving the appearance of an image morphological pro-cessing deals with tools for extracting image components that are usefulin the representation and description of shape segmentation procedurespartition an image into its constituent parts or objects object recognitionis the process that assigns a label such as vehicle to an object based onits descriptors Finally representation and description almost always fol-low the output of a segmentation stage which usually is raw pixel dataconstituting either the boundary of a region or all the points in the regionitself

and focus only on the main features (the magnetic particle in our case) Thethreshold value is usually between 0 and 255

bull The black white image which is descriped in 0 or 1 is then processed to anotherfilter from the digital image processing filters called morphological operationsexample of these operations are extrode (where you remove small white dots formthe image) diffusion (where you make the small particles bigger) opening (whereyou make first erosion then dilation) and finally closing (where the dilation isdone first then followed with erosion)

28

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

24 Kinematic Modeling of the Robotic Arm and Microparticle

Figure 212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques

bull In our case we started with erosion filter first to remove the very small particlesfom the image

bull Followed with closing to fill the gap and connect the cluster of particles so itsone object

bull The next step is to search for the objects in the image using the contour detectionmethod of openCV

bull The biggest Contour of all the contours that was detected is then selected

bull a bounding rectangle is then drawn around the object as shown in Fig3 thecenter of the rectangle relative to the image is the position of the micro particlein the image

In this work we used emguCV version 310 library in C Imaging source camera sdkfor the XY view camera and a normal webcam class to detect the YZ view using venusmicroscope The positions that was exracted from the image processing class is thenprocessed to the control function then to matlab simulink with xcontrollerycontrollerand zcontroller vector respectively

24 Kinematic Modeling of the Robotic Arm and Microparticle

Our permanent magnet-based robotic system consists of a robotic arm and a perma-nent magnet attached to its end-effector Location and orientation of the permanent

29

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

2 Experiments preparation

Figure 213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm A perma-nent magnet is attached to the end-effector and its magnetization vectorand z7 are collinear

magnet are controlled to position the microparticles in 3D space A magnetic force(F(P)) is exerted on the dipole moment (m) of a paramagnetic microparticle and en-ables pulling of this microparticle in three-dimensional space (Fig 32) The magneticforce is generated using a cylindrical NdFeB magnet (with magnetization M) with amagnetization vector oriented along y7 as shown in Fig 213

The permanent magnet is attached to the end-effector of a robotic arm (KUKA LBRiiwa 7 R800) with seven rotational DOFs Therefore the magnetic force exerted on themagnetic dipole moment of the microparticle is controlled using the orientation of thepermanent magnet and the end-effector of the robotic arm The Denavit-Hartenberg(DH) frames are shown in Fig 213 while the DH parameters are listed in Table 22

The relation between the base frame of reference of the robotic arm and a frame ofreference of a microparticle is given by

0Tp(qp) = 0T7(q) 7TM MTp(p) (222)

where 0Tp(qp) is the homogenous transformation between the frame of reference

30

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

24 Kinematic Modeling of the Robotic Arm and Microparticle

Table 22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference

i αiminus1 aiminus1 di θ

1 900 00 01875 q12 900 00 00 q23 900 00 04 q34 900 00 00 q45 -900 00435 04 q56 -900 00435 00 q6

of the robotic arm and the frame of reference of the microparticle (Fig 213) Thistransformation depends on the generalized coordinates of the robotic arm (q) and theposition of the microparticle (P) Further 0T7 represents the relation between theframe of reference of the robotic arm and the end-effector and is governed by

iminus1Ti(q) =

cθi minussθi 0 aiminus1

sθicαiminus1 cθicαiminus1 minussαiminus1 minussαiminus1disθisαiminus1 cθisαiminus1 cαiminus1 cαiminus1di

0 0 0 1

(223)

where iminus1Ti represents the relation between the i minus 1 frame and i frame of reference(0T7 = 0T1 6 T7) Further c and s indicate the cosine and sine functions respec-tively Furthermore αiminus1 aiminus1 di and θi are the DH parameters of the ith frameof reference and are provided in Table 22 In (222) 7TM represents the relationbetween the end-effector and the permanent magnet as follows

7TM =

1 0 0 00 1 0 00 0 1 lM0 0 0 1

(224)

where lM is the length of an adapter used to connect the permanent magnet to theend-effector

Finally MTp(p) describes the relation between the permanent magnet and the frame of

31

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

2 Experiments preparation

reference of the microparticle The magnetization of the permanent magnet generatesthe following magnetic field at the position of the microparticle (P) [25]

B(p) = micro0

4πxminus p3

(3 (M middot (xminus p)) (xminus p)

(xminus p) 2 minusM) (225)

In (225) B(p) is the magnetic field on point p and micro0 is the permeability of free space(4πtimes10minus7 TmAminus1) x is the position of the permanent magnet that is attached to theend-effector of the robotic arm The microparticles we consider are spherical and haveno shape anisotropy Therefore the microparticles are subjected to pure magneticforce and is given by

part

parti(m(p) middotB(p)) = fi(p) for i = x y z (226)

where m(p) is the magnetic dipole moment of the microparticle and fi(p) is the ithmagnetic force component for i = x y z In (226) the magnetic dipole moment ofthe microparticle is given by

m(p) = 1micro

43πr

3pχmB(p) (227)

where micro is the permeability coefficient given by micro = micro0(1 + χm) Further χm isthe magnetic susceptibility constant [13] and rp is the radius of the microparticleThe microparticles we use have an average diameter of 100 microm and are pulled ata maximum speed of 200 microms (speed of a single microparticle) At this speed theReynolds number is calculated as Re = ρf |v|D

η= 01 where ρf is the density of the

fluid (1260 kgmminus3) v is the velocity of the microparticle and D is its diameter andη is the dynamic viscosity of the fluid (1times10minus3 Pas)

Therefore motion of the microparticle is governed by magnetic drag and buoyancyforces as follows

part

parti(m(p) middotB(p)) + 6πηrpp + Fb = 0 for i = x y z (228)

where rp is the radius of the microparticle and p is its velocity vector η is thedynamic viscosity of the medium

32

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

24 Kinematic Modeling of the Robotic Arm and Microparticle

Fb is the net buoyancy force (effective gravitational force) and is given by

Fb = V (ρp minus ρf)g (229)

In (229) V is the volume of the microparticle and ρp is its density Further g is the ac-celeration vector due to gravity Setting the magnetic field gradient (nabla (m(P) middotB(P)))to Kpe+Kde+ Fb where Fb is the nominal buoyancy force yields the following errordynamics

Kpe + Kde + 6πηrpe = 0 (230)

where Kp and Kd are proportional and derivative positive-definite matrices respec-tively Further e and e are the position and velocity tracking errors respectively andare given by

e = PminusPr =[px py pz

]Tminus[prx pry pr

z

]T (231)

where Pr is the a fixed reference position (e = P) and prx pr

y and prz are its components

along x- y- and z-axis respectively Implementation of this control system is based onthe error dynamics of the microparticle (230) First we select positive-definite controlgains Kp and Kd and solve for the desired magnetic forces on the microparticle bysetting

Kpe + Kde + Fb rArr nabla (m(P) middotB(P)) (232)

The whole Transfromation from the base of the robot to the magnetic particles canbe expressed as shown in Fig214 Position of the microparticles is mapped onto theconfiguration of the robotic arm using three homogenous transformation The pose ofthe end-effector is determind with respect to a frame of reference using 0T 7 Positionof the permanent magnet with respect to te end-effector is defined using 7TM Positionof the microparticle with respect to the magnet is defined using MT P

33

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

2 Experiments preparation

Reference frame

End-effector

Magnet

T0 1 T

6 7

TE M

Microparticle

TM P

Figure 214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using 0T7Position of the permanent magnet with respect to the end-effector isdefined using 7TM Position of the microparticle with respect to themagnet is defined using MTp

25 Control of the Magnetic-Based Robotic System

In the area of robotics developing a control algorithm which is accurate rigid andhighly responsive mechanisms is essential for creating a robust application overcom-ing the problem of dynamical model uncertainties and its effects on the derivationof the control law through disturbance estimation and compensation [43] is the firststep towards robustness Different stduies on estimating the disturbances were donean estimation for the external torque due to the interaction of the environment in amultidegree-of-freedom systems [44] An action reaction law of dynamics the well-known disturbance observer is used to determine the incident reaction forces fromdynamical systems on the interface planes with their actuators to estimate the un-known and the inaccessible outputs [45]

Our magnetic-based robotic system the position of the paramagnetic microparticleswhich are moving in the fluid inside the glass tube is controlled using the magneticfield The model of the forces acting on the paramagnetic microparticles and thetransformation of the robotic arm is explained in the previous system

34

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

25 Control of the Magnetic-Based Robotic System

Figure 215 Closed-loop motion control of the position of a paramagnetic microparti-cle using the robotic arm (KUKA iiwa r800) holding the permanent mag-net The particles are contained in water inside a glass tube and theirposition (P) is determined using a camera system and a feature trackingsoftware The control system positions the microparticles (based on thegiven reference position (Pref) by controlling the position and orientationof the permenant magnet through the robotic arm

Closed-loop motion of the position of a paramagnetic microparticle using the roboticarm (iiwa r800) holding the permanent magnet is achieved by determining the position

35

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

2 Experiments preparation

(P) of the microparticles using camera system and a feature tracking software Thecontrol system positions the microparticles (based on the given reference position(Pref) by controlling the position and orientation of the permenant magnet throughthe robotic arm The position error (e) (229) is applied to a positive-definite gains Kp

and Kd Using (232) we calculate the magnetic field B(p) This field is substitutedin (225) to calculate the desired position of the permanent magnet (x) Finallyx is used to determine the configuration of the robotic arm (q) using its inverse-kinematics Equations (222) (225) (231) and (232) enables the permanent magnet-based robotic system to control the motion of the microparticles via the configurationof the robotic arm The first two terms in (232) represent the corrective control inputwhereas the third term is an equivalent control input This equivalent control inputdepends on the densities of the medium and the microparticles and the volume ofthe microparticles that can be determined before each trail Therefore the equivalentcontrol input is relatively accurate and results in stable linear error dynamics (230)

36

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

3

Experiments

Motion control trials are done using our permanent magnet-based robotic system(Fig 32) This system consists of a robotic arm (KUKA iiwa 7 R800 KUKA RobotAutomation Augsburg Germany) The end-effector of the robot can be controlledat accuracy of plusmn01 mm We attach a cylindrical NdFeB magnet (RL21 IBS Mag-net Berlin Germany) with diameter and hight of 20 mm and 43 mm respectivelyThe magnetization vector of the cylindrical magnet is oriented along y7 of the end-effector and generates 400 mT on its surface The end-effector approaches a reser-voir that contains a medium (density of 0857 gml) and paramagnetic microparti-cles (PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbH Rostock-Warnemuende Germany) with average diameter of 100 microm The microparticle hasmagnetization of 43 Am2kg and saturation magnetization in excess of 66 Am2kgPosition of the microparticles is observed using 2 cameras Position of the microparti-cle along xz-plane is determined using a Venus USB20 Camera whereas the positionalong xy-plane is observed using a monochrome zoom camera (DMK Z12GP031 GigEmonochrome zoom camera The Imaging Source Europe GmbH Bremen Germany)A feature tracking algorithm is used to determine the position (P) of the microparti-cles simultaneously from the xy-plane and xz-plane to calculate the position trackingerror (231)

37

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

3 Experiments

Figure 31 A permanent magnet-based actuation system is used to control and po-sition paramagnetic microparticles in a low Reynolds number fluid Thesystem consist of a robotic arm and a cylindrical NdFeB magnet attachedto its end-effector Paramagnetic microparticles with average diameter of100 microm are contained inside a glass tube

31 Position control of paramagnetic microparticles in xz-plane

First we test the ability of the permanent magnet-based robotic system to suspend themicroparticles Fig 33 shows two representative trials for controlled microparticlesmoving towards similar reference position along z-axis Fig 33(a) shows a cluster ofmicroparticles moving under the influence of its own weight and controlled magneticfield gradient The gradient is controlled via the configuration of the robotic arm andat time t=14 seconds the cluster is positioned at the reference position SimilarlyFig 33(b) shows the ability of the system to pull the same cluster upwards towardsthe reference position and at time t=18 seconds the cluster is positioned at thereference position

38

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

31 Position control of paramagnetic microparticles in xz-plane

Figure 32 A permanent magnet-based actuation system is used to control and posi-tion paramagnetic microparticles in a low Reynolds number fluid (a) Thesystem consist of a robotic arm with 7 degrees-of-freedom and a cylindricalNdFeB magnet attached to its end-effector Paramagnetic microparticles(PLAParticles-M-redF-plain from Micromod Partikeltechnologie GmbHRostock-Warnemuende Germany) with average diameter of 100 microm arecontained inside a glass tube and positioned in three-dimensional (3D)space using the field gradients (b) The configuration of the robotic arm(indicated using the joint space coordinates q) solely control the position ofthe microparticles in 3D space (c) The microparticle follows a path (bluetrajectory) controllably based on the configuration of the robotic arm P1P2 and P3 are the position vectors of the microparticles corresponding tothe robotic configurations q1 q2 and q3 respectively

311 Upward movement of the cluster

As shown in Fig 34 the rise time of the cluster during the motion upward (Dottedlines) towards the reference position As one would expect that the upward motionrsquosrise time (18 seconds) is greater that the downward motions as the cluster has toovercome its weight while moving up The average speed of the upward motion wasmeasured to be 71 mms

39

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

3 Experiments

Table 31 Setup specifications used in the experimentsSetup item Manufacturer Details

KUKA iiwa 7 R800 KUKA Robot Automa-tion Augsburg Ger-many

ndash

NdFeB magnet RL21 IBS Magnet Berlin Ger-many

diameter and hight of 20mm and 43 mm

paramagnetic microparticles PLAParticles-M-redF-plain from MicromodPartikeltechnologieGmbH Rostock- Warne-muende Germany

diameter of 100 micromMagnetization of 43Am2kg and saturationmagnetization in excessof 66 Am2kg

Camera Venus USB20 Camera -Camera DMK Z12GP031 Imaging Source Europe

GmbH Bremen Ger-many

GigE monochrome zoomcamera

312 Downward movement of the cluster

Fig 34 also shows the rise-time of the cluster during the motion downward towards thereference postion the rise time of this experiment was 14 seconds while the averagespeed was 58 mms

32 Position control of paramagnetic microparticles

we turned our attention to the motion control in 3D space We provide a referenceposition to (231) regardless to the initial position of the cluster Position of thecluster is determined using a feature tracking algorithm in the xy-plane and xz-planesimultaneously

321 Motion control trials

Fig 35 shows a representative closed-loop control result in 3D space The micropar-ticles are pulled towards the reference position after 17 seconds Figs 35(a) and (b)provide the position of the microparticles along x- and y-axis towards the reference

40

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

32 Position control of paramagnetic microparticles

Figure 33 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the KUKA LBR iiwa 7 R800 roboticarm The black arrow indicates the position of the cluster in the xy-planewhereas the blue line represents the reference position along z-axis (a) Acluster of microparticle is controlled to move downward towards a referenceposition under the influence of controlled magnetic field gradient and itsown weight The magnetic field gradient is controlled via the configurationof the robotic arm (b) The cluster is pulled up towards the referenceposition using the field gradient

position (horizontal red line) respectively The configuration of the robotic arm dur-ing this closed-loop control trial is shown in Figs 35(c d e and f) by the generalizedcoordinates qj for j = 1 7 In this trial the microparticles are pulled towards thereference position at an average speed of 165 mms and the maximum steady-stateerror is 200 microm Another setpoint tracking control result is shown in Fig 36 Thecluster is pulled towards the reference position (vertical red line) at an average speedof 143 microms as shown in Fig 36(a) The maximum position error of the micropar-ticles is 250 microm in the steady-state Position of the end-effector (and the permanentmagnet) is shown in Fig 36(b) and indicates that the robotic arm positions the mi-croparticles in approximately 47 seconds The configuration of the robotic arm during

41

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

3 Experiments

Figure 34 Representative motion control of a cluster of paramagnetic microparticlesusing a permanent magnet attached to the end-effector of the KUKA LBRiiwa 7 R800 robotic arm The microparticles is controlled to move down-ward at an average speed of 58 mms and is pulled upwards at speed of71 mms

this representative motion control trial is represented using Figs 36(c d e and f)

33 Control of paramagnetic microparticle along a trajectory

We also examine the ability of the control system to follow a trajectory as shownin Fig 37 Two reference trajectories (straight line and a sinusoidal trajectory) aredivided into waypoints that are shown by the red circles Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law (232) isimplemented based on (231) This experiment shows the ability of the permanentmagnet-based robotic system to navigate the a microparticle controllably throughoutrelatively large distance of approximately 500 body-length The maximum error is

42

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

33 Control of paramagnetic microparticle along a trajectory

Figure 35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the robotic arm(a) Position of the microparticles along x-axis The microparticles move ataverage speed of 165 mms along x-axis (b) Motion of the microparticlesalong y-axis towards the reference position at an average speed of 025mms (c d e and f) Configuration of the robotic arm during the motioncontrol of the microparticles is represented using qj for j = 1 7

calculated to be 600 microm and 300 microm for the straight line and sinusoidal trajectoriesrespectively

43

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

3 Experiments

Figure 36 Kinematic control of a paramagnetic microparticle is achieved using a per-manent magnet attached to the end-effector of the robotic arm (a) Acluster of paramagnetic microparticles is controlled in three-dimensionalspace towards a reference position (vertical red line) The cluster is pulledtowards the reference position at an average speed of 143 mms and 028mms along x- and y-axis respectively (b) Position of the end-effectorof the robotic arm is calculated during the closed-loop control of the mi-croparticles (c d e and f) Configuration of the robotic arm during thecontrol of the microparticle

44

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

33 Control of paramagnetic microparticle along a trajectory

Figure 37 Straight line and sinusoidal trajectory tracking by the controlled micropar-ticles via the controlled magnetic field gradient Waypoints (small red cir-cles) are defined along the reference trajectory (red lines) and control law(232) is implemented based on (231)

In Fig 38 we demonstrate the Joints angles of the KUKA robot during the trackingof the sinusoidal trajectory

45

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

3 Experiments

Figure 38 The configuration of the robotic arm during trajectory tracking of a sinu-soidal reference trajectory is represented using qj for j = 1 7

Finally the motion of the robot during the Sinewave experiment can be shown inFig 39

Figure 39 Representative configurations for the robotic are during the tracking of asinusoidal trajectory

The motion was limited to 8 cm due to the limitation of the field of view of thecamera Results of this experiment showed the ability of controlling the paramagnetic

46

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

33 Control of paramagnetic microparticle along a trajectory

microparticles in trajactory

47

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

4

Conclusions and Future work

Despite the nonlinearity of the applied magnetic field and field gradient our controlalgorithm achieves stable positioning of the microparticles during setpoint and trajec-tory tracking based on the derived error dynamics (230) The control law dependson corrective (based on the position and velocity errors) and equivalent (based on thenominal model of the microparticle) inputs The equivalent control inputs enable usto realize linear error dynamics of the controlled microparticle However its effec-tiveness is limited by the accuracy of the nominal model of the microparticle and theability to map the desired magnetic force into magnetic field gradients using (232)and (225) We do not also account for the microparticle-to-microparticle interactionsduring the motion control of a cluster of microparticles However Fig 33 and Fig 34demonstrate experimentally the ability of the control system to compensate for thedeviations between the equivalent control input and the actual model of the clusterresults showed that the rise time is 18 seconds and average speeds 58 mms and 71mms respectively in xz experiments Maximum steady-state error is 200 microm andaverage speed 16 mms in one of the motion control in xy-plane experiments trialswhile the other trial showed maximum steady-state error of 250 microm

49

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

4 Conclusions and Future work

41 Conclusions

We demonstrate the ability of permanent magnet-based robotic system to control themotion of paramagnetic microparticles in 3D space The microparticles are pulled con-trollably towards the reference position under the influence of magnetic field gradientsthat are solely controlled via the configuration of the robotic arm Our experimen-tal results show the ability of the system to position microparticles in 3D space withmaximum position error of 300 microm in the steady-state during setpoint tracking and600 microm during trajectory tracking Results also showed that the downward movementspeed is faster than the upward movement of the magnetic particles as the overcomethe weight while moving upward

42 Future work

As part of future studies our permanent magnet-based robotic system will be inte-grated to an ultrasound system to provide feedback instead of the visual feedbackprovided by the cameras [26] This modification will enable us to visualize and controlthe motion of magnetic drug carriers such as paramagnetic microparticles and iron-oxide nanoparticles in vivo Clusters of nanoparicles will be used as a magnetic drugcarriers owin to their low toxicity and excellent magnetic saturationMoreover themicroparticles can be controlled in a smaller tube having curves which simulate thecurvatures of veins and arteries We will also investigate the ability of the permanentmagnet-based robotic system to control the motion in 3D space in the presence ofdisturbence inputsuch as time-varying flow rates of the fluid to verify the effectiv-ness of the control techinques This modification is also necessary to mimic some ofthe phycological conditions of in vivo experimentation Finally The results can beenhanced by using non-linear control methods when controlling the position of theend-effector of the robotic arm

50

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

Bibliography

[1] R Siegel D Naishadham and A Jemal ldquoCancer statistics CA A CancerJournal for Clinicians vol 62 no 1 pp 10-29 December 2012

[2] I S M Khalil B E Wissa B G Salama and S Stramigioli ldquoWireless MotionControl of Paramagnetic Microparticles using a Magnetic-Based Robotic Systemwith an Open-Configuration In Proceedings of the International Conference onManipulation Manufacturing and Measurement on the Nanoscale (3M Nano)pp 190-196 Changchun China October 2015

[3] H Hassan M Pichel T Hageman L Abelmann I S M Khalil ldquoInflu-ence of the magnetic field on the two-dimensional control of magnetospirillumgryphiswaldense strain MSR-1 in Proceedings of the IEEE International Con-ference on Intelligent Robots and Systems (IROS) pages 5119-5124 DaejeonKorea October 2016

[4] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[5] M Elfar M Ayoub A Sameh H Abass R M Abdel-Kader I Gomaa and I SM Khalil ldquoTargeted penetration of MCF-7 cells using iron-oxide nanoparticlesin vitro in Proceedings of the IEEE RASEMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob) pages 260-265 SingaporeJune 2016

[6] M P Kummer J J Abbott B E Kartochvil R Borer A Sengul and B J Nel-son ldquoOctoMag an electromagnetic system for 5-DOF wireless micromanipula-

51

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

Bibliography

tion IEEE Transactions on Robotics vol 26 no 6 pp 1006-1017 Decem-ber 2010

[7] C Pawashe S Floyd E Diller and M Sitti ldquoTwo-dimensional autonomousmicroparticle manipulation strategies for magnetic microrobots in fluidic envi-ronments IEEE Transactions on Robotics vol 28 no 2 pp 467-477 April2012

[8] B J Nelson I K Kaliakatsos and J J Abbott ldquoMicrorobots for minimallyinvasive medicine Annual Review of Biomedical Engineering vol 12 pp 55-85April 2010

[9] M Sitti H Ceylan W Hu J Giltinan M Turan S Yim and E DillerldquoBiomedical applications of untethered mobile millimicrorobotsrdquo Proceedingsof the IEEE vol 103 no 2 pp 205-224 February 2015

[10] J Wang and W Gao ldquoNanoMicroscale Motors Biomedical Opportunities andChallengesrdquo ACS Nano vol 6 no 7 pp 5745-5751 July 2012

[11] A W Mahoney D L Cowan K M Miller and J J Abbott ldquoControl ofuntethered magnetically actuated tools using a rotating permanent magnet inany position in Proceedings of the IEEE International Conference on Roboticsand Automation (ICRA) pp 3375-3380 Minnesota USA May 2012

[12] S Martel O Felfoul J-B Mathieu A Chanu S Tamaz M Mohammadi MMankiewicz and N Tabatabaei ldquoMRI-based medical nanorobotic platform forthe control of magnetic nanoparticles and flagellated bacteria for target inter-ventions in human capillariesrdquo The International Journal of Robotics Researchvol 28 no 9 pp 1169-1182 September 2009

[13] R G McNeil R C Ritter B Wang M A Lawson G T Gillies K G WikaE G Quate M A Howard and M S Grady ldquoCharacteristics of an improvedmagnetic-implant guidance system IEEE Transactions on Biomedical Engineer-ing vol 42 no 8 pp 802-808 August 1995

[14] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculature Physical Review E vol 73 no 061919 June2006

[15] A W Mahoney and J J Abbott ldquoControl of untethered magnetically actu-ated tools with localization uncertainty using a rotating permanent magnet in

52

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

Bibliography

Proceedings of the IEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics (BioRob) pp 1632-1637 Rome Italy June 2012

[16] K E Peyer L Zhang and B J Nelson ldquoBio-inspired magnetic swimming mi-crorobots for biomedical applications Nanoscale vol 5 no 4 pp 1259-1272November 2012

[17] D J Bell S Leutenegger K M Hammar L X Dong and B J NelsonldquoFlagella-like propulsion for microrobots using a magnetic nanocoil and a rotat-ing electromagnetic field in Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA) pp 1128-1133 April 2007

[18] N D Nelson and J J Abbott ldquoGenerating two independent rotating magneticfields with a single magnetic dipole for the propulsion of untethered magneticdevices in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) pp 4056ndash4061 Seattle Washington USA May 2005

[19] A W Mahoney and Jake J Abbott ldquoFive-degree-of-freedom manipulation ofan untethered magnetic device in fluid using a single permanent magnet withapplication in stomach capsule endoscopy The International Journal of RoboticsResearch vol 35 no 1-3 pp 129ndash147 February 2016

[20] A Hosney A Klingner S Misra and I S M Khalil ldquoPropulsion and steer-ing of helical magnetic microrobots using two synchronized rotating dipole fieldsin three-dimensional space in Proceedings of the IEEERSJ International Con-ference of Robotics and Systems (IROS) pp 1988ndash1993 Hamburg GermanyNovember 2015

[21] I S M Khalil A F Tabak K Sadek D Mahdy N Hamdi and M Sitti ldquoRub-bing against blood clots using helical robots modeling and in vitro experimentalvalidation IEEE Robotics and Automation Letters vol 2 no 2 pp 927ndash934April 2017

[22] I S M Khalil Y Michel B Su and S Misra ldquoFeeling paramagnetic micro-particles trapped inside gas bubbles a tele-manipulation study in Proceedingsof the the International Conference on Manipulation Manufacturing and Mea-surement on the Nanoscale (3M Nano) pp 225 Chongqing China July 2016

[23] A El-Gazzar L Al-khouly A Klingner S Misra and I S M Khalil ldquoNon-contact manipulation of microbeads via pushing and pulling using magneticallycontrolled clusters of paramagnetic microparticles in Proceedings of the IEEE

53

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

Bibliography

International Conference on Intelligent Robots and Systems (IROS) HamburgGermany pp 778-783 September-October 2015

[24] K Youakim M Ehab O Hatem S Misra and I S M Khalil ldquoParamag-netic microparticles sliding on a surface characterization and closed-loop motioncontrol in Proceedings of the IEEE International Conference on Robotics andAutomation (ICRA) Seattle USA pp 4068-4073 May-June 2015

[25] E P Furlani and K C Ng ldquoAnalytical model of magnetic nanoparticle transportand capture in the microvasculaturerdquo Physical Review E vol 73 no 061919 June2006

[26] S Tognarelli V Castelli G Ciuti C Di Natali E Sinibaldi P Dario and AMenciassi ldquoMagnetic propulsion and ultrasound tracking of endovascular devicesrdquoJournal of Robotic Surgery vol 6 no 1 pp 5ndash12 December 2012

[27] I S M Khalil L Abelmann and S Misra ldquoMagnetic based motion control ofparamagnetic microparticles with disturbance compensation IEEE Transactionson Magnetics vol 50 no 10 October 2014

[28] Da vinci robot httpswwwintuitivesurgicalcomproductsdavinci_surgical_system

[29] N Nakamura N Sugano T Nishii A Kakimoto and H Miki ldquoA comparisonbetween robotic-assisted and manual implantation of cementless total hip arthro-plasty Clin Orthop Relat Res pp 1072-1081 November 2010

[30] S Nishihara N Sugano T Nishii H Miki N Nakamura and H YoshikawaldquoComparison between hand rasping and robotic milling for stem implantation incementless total hip arthroplasty Journal Arthroplasty pp 957-966 October2006

[31] W Barger A Bauer and M Borner ldquoPrimary and revision total hip replacementusing the Robodoc system Clin Orthop Relat Res pp 82-91 September 1998

[32] E Song J Seon and J Yim ldquoRobotic-assisted TKA Reduces PostoperativeAlignment Outliers and Improves Gap Balance Compared to Conventional TKAClin Orthop Relat Res Epub June 2012

[33] K Hagio N Sugano M Takashina T Nishii H Yoshikawa and T Ochi ldquoEf-fectiveness of the ROBODOC system in preventing intraoperative pulmonaryembolism Acta Orthop Scand pp 264-269 July 2003

54

Bibliography

[34] H Abbass M S Shoukry S Misra and I S M Khalil ldquoRobust motion controlof paramagnetic micro-particles against time-varying flow rates in Proceedingsof the IEEE RASEMBS International Conference on Biomedical Robotics andBiomechatronics (BioRob) pp 67-72 Singapore June 2016

[35] I S M Khalil V Magdanz S Sanchez O G Schmidt S Misra ldquoPrecise local-ization and control of catalytic janus micromotors using weak magnetic fieldsInternational Journal of Advanced Robotic Systems vol 12 no 2 January 2015

[36] I S M Khalil R M P Metz B A Reefman and S Misra ldquoMagnetic-based minimum input motion control of paramagnetic microparticles in three-dimensional space in Proceedings of the IEEE International Conference on In-telligent Robots and Systems (IROS) Tokyo Japan pp 2053-2058 November2013

[37] H Zhuang S Roth ldquoCamera-Aided Robot Calibration June 1996

[38] T Clarke C Fryer ldquoThe Development of Camera Calibration Methods andModels In emphPhotogrammetric Record 16(91) pp 51-66 April 1998

[39] P Azad T Gockel R Dillmann ldquoComputer Vision Das Praxisbuch Elektor-Verlag Aachen Mai 2009

[40] Homepage Institut fur Bildverarbeitung der ETH Zurich httpwwwvisioneeethzchsoftwarecalibration_toolboxcalibration_toolboxphpStand15112010]

[41] Udp protocol in csharp httpsmsdnmicrosoftcomen-uslibrarysystemnetsocketsudpclient28v=vs11029aspxf=255ampMSPPError=-2147217396

[42] Udp protocol in java httpsdocsoraclecomjavasetutorialnetworkingdatagramsclientServerhtml

[43] I S M Khalil E Globovic and A Sabanovic ldquoHigh precision motion control ofparallel robots with imperfections and manufacturing tolerances in Proceedingsof the IEEE International Conference on Mechatronics (ICM) Istanbul Turkeypp 39-44 April 2011

[44] I S M Khalil and A Sabanovic ldquoAction-reaction-based parameters identifica-tion and states estimation of flexible systems in Proceedings of the IEEE Inter-

55

Bibliography

national Conference on Industrial Technology (ISIE) Pari Italy pp 46-51 July2010

[45] I S M Khalil and A Sabanovic ldquoSensorless action-reaction-based residual vibra-tion suppression for multi-degree-of-freedom flexible systems in Proceedings ofthe IEEE International Conference on Industrial Technology (IECON) PhoenixUSA pp 1627-1632 November 2010

[46] I S M Khalil and A Sabanovic ldquoSensorless torque estimation in multidegree-of-freedom flexible system in Proceedings of the IEEE International Conferenceon Industrial Technology (IECON) Porto Portugal pp 2354-2359 November2009

56

List of Abbreviations

e speed error

η Dynamic viscosity of the medium

micro permeability coefficient

micro0 Permeability of free space

π Math PI 314

ρf Density of medium

ρp Density of microparticle

cHw Transformation from camera to world frame

cP Camera point

IFHc Transformation from Image frame to camera

IHc Transformation from Image to camera frame

IHIF Transformation from Image to Image frame

57

Bibliography

IHw Transformation from Image to world frame

IP Image point

wHc Transformation from world to camera frame

wP World point

cα Cosine alpha is part of the DH parameter

cθ Cosine Theta is part of the DH parameter

Fb Buoyancy force

fi Magnetic force component along xy z respectively

g Acceleration vector due to gravity

KD Derivative Gain

KI Integral Gain

KP Proportional Gain

lm Length of the permanent magnet

m(P ) Magnetic dipole moment

q Joints angles of the kuka iiwa r800

R Rotation Matrix

rp Radius of particle

sα Sine alpha is part of the DH parameter

58

Bibliography

sθ Sine Theta is part of the DH parameter

stx1 Selected triangle vertices 1 in X direction

stx2 Selected triangle vertices 2 in X direction

stx3 Selected triangle vertices 3 in X direction

sty1 Selected triangle vertices 1 in y direction

sty2 Selected triangle vertices 2 in y direction

sty3 Selected triangle vertices 3 in y direction

stz1 Selected triangle vertices 1 in z direction

stz2 Selected triangle vertices 2 in z direction

stz3 Selected triangle vertices 3 in z direction

T Transformation matrix

v Volume of microparticle

Xm Magnetic susceptibility constant

B(P) Magnetic field with respect to particle position

DH Parameter denavit hartenberg parameter

e Position error

F(P) Magnetic Force with respect to the particle position

M Magnetization of the permanent magnet

59

Bibliography

TC Selected triangle Center

TCX Selected triangle Center in X direction

TCY Selected triangle Center in Y direction

TCZ Selected triangle Center in Z direction

60

List of Figures

11 The OctoMag prototype is designed and constructed at ETH ZurichThe system contains eight electromagnets in closed configuration [6] 6

12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltech-nologie GmbH Rostock-Warnemuende Germany) in three-dimensionalspace The magnetic system consists of 8 iron-core electromagnets thatsurrounds a water reservoir Magnetic-based robotic system give morefreedom in moving through trajectories compared to this system How-ever this system showed greater positioning accuracy compared to themagnet-based one [36] 7

21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console wherethe surgeon sits while operating a patient-side cart where the patient ispositioned during surgery interactive robotic arms a three-dimensionalhigh definition vision system and proprietary EndoWrist instruments[28] (a) The EndoWrist Insturment (b) Surgen console (c) visionsystem (d) Patient-side cart 13

22 ROBODOC Surgical system that enables pre-surgical 3D planningMeanwhile the ROBODOC Surgical assistant executes the pre-surgicalplan with unparalleled precision Benefits of the ROBODOC Systemover traditional methods include improved cavity fit fill and alignmentfor prosthetic implants [29] [30] [32] as well as a reduction of leg lengthdiscrepancy [30] and intra-operative fractures [29] [31] 14

23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom 15

61

List of Figures

24 Communication protocol architecture consisting of 5 layers It startswith the physical layer that deals with the physical (ie electricaland mechanical) aspects of transmitting data (eg voltage levels pin-connector design cable lengths and grounding arrangements) the Data-link layer deals with the transmission of data frames (eg packets) overa physical link between network entities including the incorporation oferror-correction coding into the data frames the network that dealswith establishing paths for data between a pair of computers and han-dling any switching among alternative routes between the computersas well as with definitions of how to break messages (or packets) upinto individual packets of data in such a way that the packets canbe transmitted and then reassembled Transport layer deals with datatransfer between end systems flow control for two computers Finallyapplication layer that deals with the interface between a user and thehost computer 16

25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero 17

26 Aimed position on a selected triangle in a Computer Aided Device file 19

27 Point calibration grid with orientation beam used to calibrate the cam-eras and extract the intrinsic parameters 20

28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is thestraight line which is perpendicular to the image plane and through theprojection center becomes optical axis CameraP point on the cameraframe ImageP point in the image frame wHc is the transformationfrom the world to camera frame f focal length C is the image startingpoint and it is the interaction between the optical axis and the imageplane 22

29 This image show the yz-view and the xz-view of the camera In theyz-plane The CPy point is mapped to PPv using a beamset 23

210 The point calibration grid with orientation beam is implemented toextract the intrinsic parameters of the camera [40] 26

62

List of Figures

211 Digital image processing steps image acquisition is the first step orprocess of the fundamental steps of digital image processing Imageacquisition could be as simple as being given an image that is alreadyin digital form image enhancement is among the simplest and mostappealing areas of digital image processing The idea behind enhance-ment techniques is to bring out detail that is obscured image restortionis an area that also deals with improving the appearance of an imagemorphological processing deals with tools for extracting image compo-nents that are useful in the representation and description of shapesegmentation procedures partition an image into its constituent partsor objects object recognition is the process that assigns a label suchas vehicle to an object based on its descriptors Finally representa-tion and description almost always follow the output of a segmentationstage which usually is raw pixel data constituting either the boundaryof a region or all the points in the region itself 28

212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques 29

213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm Apermanent magnet is attached to the end-effector and its magnetizationvector and z7 are collinear 30

214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using0T7 Position of the permanent magnet with respect to the end-effectoris defined using 7TM Position of the microparticle with respect to themagnet is defined using MTp 34

215 Closed-loop motion control of the position of a paramagnetic micropar-ticle using the robotic arm (KUKA iiwa r800) holding the permanentmagnet The particles are contained in water inside a glass tube andtheir position (P) is determined using a camera system and a fea-ture tracking software The control system positions the microparticles(based on the given reference position (Pref) by controlling the positionand orientation of the permenant magnet through the robotic arm 35

31 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluidThe system consist of a robotic arm and a cylindrical NdFeB magnetattached to its end-effector Paramagnetic microparticles with averagediameter of 100 microm are contained inside a glass tube 38

63

List of Figures

32 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluid(a) The system consist of a robotic arm with 7 degrees-of-freedom anda cylindrical NdFeB magnet attached to its end-effector Paramag-netic microparticles (PLAParticles-M-redF-plain from Micromod Par-tikeltechnologie GmbH Rostock-Warnemuende Germany) with aver-age diameter of 100 microm are contained inside a glass tube and posi-tioned in three-dimensional (3D) space using the field gradients (b)The configuration of the robotic arm (indicated using the joint spacecoordinates q) solely control the position of the microparticles in 3Dspace (c) The microparticle follows a path (blue trajectory) control-lably based on the configuration of the robotic arm P1 P2 and P3 arethe position vectors of the microparticles corresponding to the roboticconfigurations q1 q2 and q3 respectively 39

33 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the KUKA LBR iiwa 7R800 robotic arm The black arrow indicates the position of the clusterin the xy-plane whereas the blue line represents the reference posi-tion along z-axis (a) A cluster of microparticle is controlled to movedownward towards a reference position under the influence of controlledmagnetic field gradient and its own weight The magnetic field gradientis controlled via the configuration of the robotic arm (b) The clusteris pulled up towards the reference position using the field gradient 41

34 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the end-effector of theKUKA LBR iiwa 7 R800 robotic arm The microparticles is controlledto move downward at an average speed of 58 mms and is pulledupwards at speed of 71 mms 42

35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the roboticarm (a) Position of the microparticles along x-axis The microparticlesmove at average speed of 165 mms along x-axis (b) Motion of themicroparticles along y-axis towards the reference position at an averagespeed of 025 mms (c d e and f) Configuration of the robotic armduring the motion control of the microparticles is represented using qjfor j = 1 7 43

64

List of Figures

36 Kinematic control of a paramagnetic microparticle is achieved usinga permanent magnet attached to the end-effector of the robotic arm(a) A cluster of paramagnetic microparticles is controlled in three-dimensional space towards a reference position (vertical red line) Thecluster is pulled towards the reference position at an average speed of143 mms and 028 mms along x- and y-axis respectively (b) Po-sition of the end-effector of the robotic arm is calculated during theclosed-loop control of the microparticles (c d e and f) Configurationof the robotic arm during the control of the microparticle 44

37 Straight line and sinusoidal trajectory tracking by the controlled mi-croparticles via the controlled magnetic field gradient Waypoints (smallred circles) are defined along the reference trajectory (red lines) andcontrol law (232) is implemented based on (231) 45

38 The configuration of the robotic arm during trajectory tracking of asinusoidal reference trajectory is represented using qj for j = 1 7 46

39 Representative configurations for the robotic are during the tracking ofa sinusoidal trajectory 46

1 Sending message through User Datagram Protocol(UDP) Consists of amessage of 192 bytes that holds the information about the robot stateFirst 24 byts show the cartesian stiffness then 28 bytes that show theaxis stiffness followd by 24 bytes that contains the cartesian dampingthen another 28 bytes with the axis damping information Further 28bytes with the Joints positions values followed by 24 bytes with thecartesian positions Then 4 bytes for the position flag and the messageends with 32 bytes that shows the digital outputs state 70

2 Receiving a message through User Datagram Protocol(UDP) consist-ing of 192 bytes that hold the information about the robot state first28 bytes show the information of the joings angles the next 24 bytesthat show the cartesian position followd by 28 bytes that contains thetorques of the joints then another 28 bytes with the external torquesfurther 12 bytes with the external forces followed by 8 bytes with thetimestamps Then 32 bytes for the digital inputs and the message endswith 32 bytes that shows the digital outputs state 71

3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras 74

65

List of Tables

21 This table shows the definitions of the intrinsic parameters that areextracted 21

22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ithframe of reference 31

31 Setup specifications used in the experiments 40

67

Appendix

1 Communication protocol User Datagram Protocol (UDP)

In this thesis we use UDP as our mean of communication between C and iiwa server(uses java as programming language) the reason behind choosing UDP is because wewanted to transfer the data biderectionally as fast as possible With UDP we cantransfer data in the optimal scenario every 4 millisecond wheras in TCP data thefastest we can get out of the protocol is 8msAn example of how to create a UDPcommunication is shown in the appendix

1001 Sending bytes message from robot server to C

Every 4 ms the windows application which acts as a server in this case sends 192 bytesmessage to the robot server which takes the role of the client in this case The messageis divided into sections as shown in the Figure 1 An example is shown in the appendixto explain how to send byte messages from the robot server or from the c application

The first 24 bytes contains details about the Cartesian stiffness the first 4 bytes aredecoded afterward to extract the stiffness value in X the 4 follow bytes decoded tostiffness value in Y then the next 4 decoded to stiffness value in Z then stiffness valuein A stiffness value in B and finally the stiffness value in C The next 28 bytes holdthe details of the 7 axis stiffness each 4 bytes are decoded to extract the stiffness ofA1 A2 A3 A4 A5 A6 A7 respectively Followed with another 24 bytes which holddetails about Cartesian damping each 4 bytes are decoded to extract the float values

69

List of Tables

Figure 1 Sending message through User Datagram Protocol(UDP) Consists of a mes-sage of 192 bytes that holds the information about the robot state First 24byts show the cartesian stiffness then 28 bytes that show the axis stiffnessfollowd by 24 bytes that contains the cartesian damping then another 28bytes with the axis damping information Further 28 bytes with the Jointspositions values followed by 24 bytes with the cartesian positions Then 4bytes for the position flag and the message ends with 32 bytes that showsthe digital outputs state

of damping in X Y Z A B C respectively Followed with 28 bytes which hold detailsabout Axis damping each 4 bytes are decoded to extract the float values of dampingin A1 A2 A3 A4 A5 A6 A7 respectively Followed with another 28 bytes whichhold details about joint positions each 4 bytes are decoded to extract the float valuesof joint angle (in radian) in A1 A2 A3 A4 A5 A6 A7 respectively Followed with24 bytes which hold details about Cartesian positions each 4 bytes are decoded toextract the float values of XY and Z (in mm) followed by A B and C (in radian)respectively Followed with 4 bytes that is decoded to a float number between 1 and4 defined as follows

1 No movement2 Gravity compensation movement3 Axis movement4 Cartesian movement

Finally the last 32 bytes are used to transfer the values of digital outputs each 2 bytesare decoded to give an integer value either 0 or 1 for each output of the 16 outputs

0 Output of turned off1 Output of turned on

70

1 Communication protocol User Datagram Protocol (UDP)

1002 Receiving byte message from C application in the robot serverOn a different thread every 4 ms the Robot server which acts as a server in this casesends 192 bytes message to the windows application which takes the role of the clientin this case The message is divided into sections as shown in the Figure 2 An exam-ple of how to receive message in c and in the robot message is shown in the appendix

Figure 2 Receiving a message through User Datagram Protocol(UDP) consisting of192 bytes that hold the information about the robot state first 28 bytesshow the information of the joings angles the next 24 bytes that show thecartesian position followd by 28 bytes that contains the torques of the jointsthen another 28 bytes with the external torques further 12 bytes with theexternal forces followed by 8 bytes with the timestamps Then 32 bytes forthe digital inputs and the message ends with 32 bytes that shows the digitaloutputs state

The First 28 bytes shows the Joint Positions These bytes are decoded as explained inthe sending paragraph first 4 bytes have Joints position (in Radian) for A1 A2 A3A4 A5 A6 A7 respectively Followed by 24 bytes that shows the measured Cartesianposition X Y Z (in mm) then A B C (in Radian) respectively Followed by 28 bytesthat contains the measured axis torques (in Nm) for A1 A2 A3 A4 A5 A6 finallyA7 respectively Followed by another 28 bytes that shows the external axis torques(in Nm) for A1 A2 A3 A4 A5 A6 A7 respectively Then 12 bytes that showsthe external forces (in Newton) for Cartesian X Y and Z respectively Followed by 4bytes that is decoded to float that shows the measured torque timestamp (in seconds)Followed by another 4 bytes that is decoded also to another float that shows externaltorque timestamp (in seconds) The last two packets of data contain the details ofthe digital inputsoutputs the first 32 bytes are for 16 digital inputs each 2 bytes aredecoded into an integer of float that is either 0 or 10 Input of turn off1 Input of turn on

71

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

List of Abbreviations

e speed error

η Dynamic viscosity of the medium

micro permeability coefficient

micro0 Permeability of free space

π Math PI 314

ρf Density of medium

ρp Density of microparticle

cHw Transformation from camera to world frame

cP Camera point

IFHc Transformation from Image frame to camera

IHc Transformation from Image to camera frame

IHIF Transformation from Image to Image frame

57

Bibliography

IHw Transformation from Image to world frame

IP Image point

wHc Transformation from world to camera frame

wP World point

cα Cosine alpha is part of the DH parameter

cθ Cosine Theta is part of the DH parameter

Fb Buoyancy force

fi Magnetic force component along xy z respectively

g Acceleration vector due to gravity

KD Derivative Gain

KI Integral Gain

KP Proportional Gain

lm Length of the permanent magnet

m(P ) Magnetic dipole moment

q Joints angles of the kuka iiwa r800

R Rotation Matrix

rp Radius of particle

sα Sine alpha is part of the DH parameter

58

Bibliography

sθ Sine Theta is part of the DH parameter

stx1 Selected triangle vertices 1 in X direction

stx2 Selected triangle vertices 2 in X direction

stx3 Selected triangle vertices 3 in X direction

sty1 Selected triangle vertices 1 in y direction

sty2 Selected triangle vertices 2 in y direction

sty3 Selected triangle vertices 3 in y direction

stz1 Selected triangle vertices 1 in z direction

stz2 Selected triangle vertices 2 in z direction

stz3 Selected triangle vertices 3 in z direction

T Transformation matrix

v Volume of microparticle

Xm Magnetic susceptibility constant

B(P) Magnetic field with respect to particle position

DH Parameter denavit hartenberg parameter

e Position error

F(P) Magnetic Force with respect to the particle position

M Magnetization of the permanent magnet

59

Bibliography

TC Selected triangle Center

TCX Selected triangle Center in X direction

TCY Selected triangle Center in Y direction

TCZ Selected triangle Center in Z direction

60

List of Figures

11 The OctoMag prototype is designed and constructed at ETH ZurichThe system contains eight electromagnets in closed configuration [6] 6

12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltech-nologie GmbH Rostock-Warnemuende Germany) in three-dimensionalspace The magnetic system consists of 8 iron-core electromagnets thatsurrounds a water reservoir Magnetic-based robotic system give morefreedom in moving through trajectories compared to this system How-ever this system showed greater positioning accuracy compared to themagnet-based one [36] 7

21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console wherethe surgeon sits while operating a patient-side cart where the patient ispositioned during surgery interactive robotic arms a three-dimensionalhigh definition vision system and proprietary EndoWrist instruments[28] (a) The EndoWrist Insturment (b) Surgen console (c) visionsystem (d) Patient-side cart 13

22 ROBODOC Surgical system that enables pre-surgical 3D planningMeanwhile the ROBODOC Surgical assistant executes the pre-surgicalplan with unparalleled precision Benefits of the ROBODOC Systemover traditional methods include improved cavity fit fill and alignmentfor prosthetic implants [29] [30] [32] as well as a reduction of leg lengthdiscrepancy [30] and intra-operative fractures [29] [31] 14

23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom 15

61

List of Figures

24 Communication protocol architecture consisting of 5 layers It startswith the physical layer that deals with the physical (ie electricaland mechanical) aspects of transmitting data (eg voltage levels pin-connector design cable lengths and grounding arrangements) the Data-link layer deals with the transmission of data frames (eg packets) overa physical link between network entities including the incorporation oferror-correction coding into the data frames the network that dealswith establishing paths for data between a pair of computers and han-dling any switching among alternative routes between the computersas well as with definitions of how to break messages (or packets) upinto individual packets of data in such a way that the packets canbe transmitted and then reassembled Transport layer deals with datatransfer between end systems flow control for two computers Finallyapplication layer that deals with the interface between a user and thehost computer 16

25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero 17

26 Aimed position on a selected triangle in a Computer Aided Device file 19

27 Point calibration grid with orientation beam used to calibrate the cam-eras and extract the intrinsic parameters 20

28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is thestraight line which is perpendicular to the image plane and through theprojection center becomes optical axis CameraP point on the cameraframe ImageP point in the image frame wHc is the transformationfrom the world to camera frame f focal length C is the image startingpoint and it is the interaction between the optical axis and the imageplane 22

29 This image show the yz-view and the xz-view of the camera In theyz-plane The CPy point is mapped to PPv using a beamset 23

210 The point calibration grid with orientation beam is implemented toextract the intrinsic parameters of the camera [40] 26

62

List of Figures

211 Digital image processing steps image acquisition is the first step orprocess of the fundamental steps of digital image processing Imageacquisition could be as simple as being given an image that is alreadyin digital form image enhancement is among the simplest and mostappealing areas of digital image processing The idea behind enhance-ment techniques is to bring out detail that is obscured image restortionis an area that also deals with improving the appearance of an imagemorphological processing deals with tools for extracting image compo-nents that are useful in the representation and description of shapesegmentation procedures partition an image into its constituent partsor objects object recognition is the process that assigns a label suchas vehicle to an object based on its descriptors Finally representa-tion and description almost always follow the output of a segmentationstage which usually is raw pixel data constituting either the boundaryof a region or all the points in the region itself 28

212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques 29

213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm Apermanent magnet is attached to the end-effector and its magnetizationvector and z7 are collinear 30

214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using0T7 Position of the permanent magnet with respect to the end-effectoris defined using 7TM Position of the microparticle with respect to themagnet is defined using MTp 34

215 Closed-loop motion control of the position of a paramagnetic micropar-ticle using the robotic arm (KUKA iiwa r800) holding the permanentmagnet The particles are contained in water inside a glass tube andtheir position (P) is determined using a camera system and a fea-ture tracking software The control system positions the microparticles(based on the given reference position (Pref) by controlling the positionand orientation of the permenant magnet through the robotic arm 35

31 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluidThe system consist of a robotic arm and a cylindrical NdFeB magnetattached to its end-effector Paramagnetic microparticles with averagediameter of 100 microm are contained inside a glass tube 38

63

List of Figures

32 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluid(a) The system consist of a robotic arm with 7 degrees-of-freedom anda cylindrical NdFeB magnet attached to its end-effector Paramag-netic microparticles (PLAParticles-M-redF-plain from Micromod Par-tikeltechnologie GmbH Rostock-Warnemuende Germany) with aver-age diameter of 100 microm are contained inside a glass tube and posi-tioned in three-dimensional (3D) space using the field gradients (b)The configuration of the robotic arm (indicated using the joint spacecoordinates q) solely control the position of the microparticles in 3Dspace (c) The microparticle follows a path (blue trajectory) control-lably based on the configuration of the robotic arm P1 P2 and P3 arethe position vectors of the microparticles corresponding to the roboticconfigurations q1 q2 and q3 respectively 39

33 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the KUKA LBR iiwa 7R800 robotic arm The black arrow indicates the position of the clusterin the xy-plane whereas the blue line represents the reference posi-tion along z-axis (a) A cluster of microparticle is controlled to movedownward towards a reference position under the influence of controlledmagnetic field gradient and its own weight The magnetic field gradientis controlled via the configuration of the robotic arm (b) The clusteris pulled up towards the reference position using the field gradient 41

34 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the end-effector of theKUKA LBR iiwa 7 R800 robotic arm The microparticles is controlledto move downward at an average speed of 58 mms and is pulledupwards at speed of 71 mms 42

35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the roboticarm (a) Position of the microparticles along x-axis The microparticlesmove at average speed of 165 mms along x-axis (b) Motion of themicroparticles along y-axis towards the reference position at an averagespeed of 025 mms (c d e and f) Configuration of the robotic armduring the motion control of the microparticles is represented using qjfor j = 1 7 43

64

List of Figures

36 Kinematic control of a paramagnetic microparticle is achieved usinga permanent magnet attached to the end-effector of the robotic arm(a) A cluster of paramagnetic microparticles is controlled in three-dimensional space towards a reference position (vertical red line) Thecluster is pulled towards the reference position at an average speed of143 mms and 028 mms along x- and y-axis respectively (b) Po-sition of the end-effector of the robotic arm is calculated during theclosed-loop control of the microparticles (c d e and f) Configurationof the robotic arm during the control of the microparticle 44

37 Straight line and sinusoidal trajectory tracking by the controlled mi-croparticles via the controlled magnetic field gradient Waypoints (smallred circles) are defined along the reference trajectory (red lines) andcontrol law (232) is implemented based on (231) 45

38 The configuration of the robotic arm during trajectory tracking of asinusoidal reference trajectory is represented using qj for j = 1 7 46

39 Representative configurations for the robotic are during the tracking ofa sinusoidal trajectory 46

1 Sending message through User Datagram Protocol(UDP) Consists of amessage of 192 bytes that holds the information about the robot stateFirst 24 byts show the cartesian stiffness then 28 bytes that show theaxis stiffness followd by 24 bytes that contains the cartesian dampingthen another 28 bytes with the axis damping information Further 28bytes with the Joints positions values followed by 24 bytes with thecartesian positions Then 4 bytes for the position flag and the messageends with 32 bytes that shows the digital outputs state 70

2 Receiving a message through User Datagram Protocol(UDP) consist-ing of 192 bytes that hold the information about the robot state first28 bytes show the information of the joings angles the next 24 bytesthat show the cartesian position followd by 28 bytes that contains thetorques of the joints then another 28 bytes with the external torquesfurther 12 bytes with the external forces followed by 8 bytes with thetimestamps Then 32 bytes for the digital inputs and the message endswith 32 bytes that shows the digital outputs state 71

3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras 74

65

List of Tables

21 This table shows the definitions of the intrinsic parameters that areextracted 21

22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ithframe of reference 31

31 Setup specifications used in the experiments 40

67

Appendix

1 Communication protocol User Datagram Protocol (UDP)

In this thesis we use UDP as our mean of communication between C and iiwa server(uses java as programming language) the reason behind choosing UDP is because wewanted to transfer the data biderectionally as fast as possible With UDP we cantransfer data in the optimal scenario every 4 millisecond wheras in TCP data thefastest we can get out of the protocol is 8msAn example of how to create a UDPcommunication is shown in the appendix

1001 Sending bytes message from robot server to C

Every 4 ms the windows application which acts as a server in this case sends 192 bytesmessage to the robot server which takes the role of the client in this case The messageis divided into sections as shown in the Figure 1 An example is shown in the appendixto explain how to send byte messages from the robot server or from the c application

The first 24 bytes contains details about the Cartesian stiffness the first 4 bytes aredecoded afterward to extract the stiffness value in X the 4 follow bytes decoded tostiffness value in Y then the next 4 decoded to stiffness value in Z then stiffness valuein A stiffness value in B and finally the stiffness value in C The next 28 bytes holdthe details of the 7 axis stiffness each 4 bytes are decoded to extract the stiffness ofA1 A2 A3 A4 A5 A6 A7 respectively Followed with another 24 bytes which holddetails about Cartesian damping each 4 bytes are decoded to extract the float values

69

List of Tables

Figure 1 Sending message through User Datagram Protocol(UDP) Consists of a mes-sage of 192 bytes that holds the information about the robot state First 24byts show the cartesian stiffness then 28 bytes that show the axis stiffnessfollowd by 24 bytes that contains the cartesian damping then another 28bytes with the axis damping information Further 28 bytes with the Jointspositions values followed by 24 bytes with the cartesian positions Then 4bytes for the position flag and the message ends with 32 bytes that showsthe digital outputs state

of damping in X Y Z A B C respectively Followed with 28 bytes which hold detailsabout Axis damping each 4 bytes are decoded to extract the float values of dampingin A1 A2 A3 A4 A5 A6 A7 respectively Followed with another 28 bytes whichhold details about joint positions each 4 bytes are decoded to extract the float valuesof joint angle (in radian) in A1 A2 A3 A4 A5 A6 A7 respectively Followed with24 bytes which hold details about Cartesian positions each 4 bytes are decoded toextract the float values of XY and Z (in mm) followed by A B and C (in radian)respectively Followed with 4 bytes that is decoded to a float number between 1 and4 defined as follows

1 No movement2 Gravity compensation movement3 Axis movement4 Cartesian movement

Finally the last 32 bytes are used to transfer the values of digital outputs each 2 bytesare decoded to give an integer value either 0 or 1 for each output of the 16 outputs

0 Output of turned off1 Output of turned on

70

1 Communication protocol User Datagram Protocol (UDP)

1002 Receiving byte message from C application in the robot serverOn a different thread every 4 ms the Robot server which acts as a server in this casesends 192 bytes message to the windows application which takes the role of the clientin this case The message is divided into sections as shown in the Figure 2 An exam-ple of how to receive message in c and in the robot message is shown in the appendix

Figure 2 Receiving a message through User Datagram Protocol(UDP) consisting of192 bytes that hold the information about the robot state first 28 bytesshow the information of the joings angles the next 24 bytes that show thecartesian position followd by 28 bytes that contains the torques of the jointsthen another 28 bytes with the external torques further 12 bytes with theexternal forces followed by 8 bytes with the timestamps Then 32 bytes forthe digital inputs and the message ends with 32 bytes that shows the digitaloutputs state

The First 28 bytes shows the Joint Positions These bytes are decoded as explained inthe sending paragraph first 4 bytes have Joints position (in Radian) for A1 A2 A3A4 A5 A6 A7 respectively Followed by 24 bytes that shows the measured Cartesianposition X Y Z (in mm) then A B C (in Radian) respectively Followed by 28 bytesthat contains the measured axis torques (in Nm) for A1 A2 A3 A4 A5 A6 finallyA7 respectively Followed by another 28 bytes that shows the external axis torques(in Nm) for A1 A2 A3 A4 A5 A6 A7 respectively Then 12 bytes that showsthe external forces (in Newton) for Cartesian X Y and Z respectively Followed by 4bytes that is decoded to float that shows the measured torque timestamp (in seconds)Followed by another 4 bytes that is decoded also to another float that shows externaltorque timestamp (in seconds) The last two packets of data contain the details ofthe digital inputsoutputs the first 32 bytes are for 16 digital inputs each 2 bytes aredecoded into an integer of float that is either 0 or 10 Input of turn off1 Input of turn on

71

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

Bibliography

IHw Transformation from Image to world frame

IP Image point

wHc Transformation from world to camera frame

wP World point

cα Cosine alpha is part of the DH parameter

cθ Cosine Theta is part of the DH parameter

Fb Buoyancy force

fi Magnetic force component along xy z respectively

g Acceleration vector due to gravity

KD Derivative Gain

KI Integral Gain

KP Proportional Gain

lm Length of the permanent magnet

m(P ) Magnetic dipole moment

q Joints angles of the kuka iiwa r800

R Rotation Matrix

rp Radius of particle

sα Sine alpha is part of the DH parameter

58

Bibliography

sθ Sine Theta is part of the DH parameter

stx1 Selected triangle vertices 1 in X direction

stx2 Selected triangle vertices 2 in X direction

stx3 Selected triangle vertices 3 in X direction

sty1 Selected triangle vertices 1 in y direction

sty2 Selected triangle vertices 2 in y direction

sty3 Selected triangle vertices 3 in y direction

stz1 Selected triangle vertices 1 in z direction

stz2 Selected triangle vertices 2 in z direction

stz3 Selected triangle vertices 3 in z direction

T Transformation matrix

v Volume of microparticle

Xm Magnetic susceptibility constant

B(P) Magnetic field with respect to particle position

DH Parameter denavit hartenberg parameter

e Position error

F(P) Magnetic Force with respect to the particle position

M Magnetization of the permanent magnet

59

Bibliography

TC Selected triangle Center

TCX Selected triangle Center in X direction

TCY Selected triangle Center in Y direction

TCZ Selected triangle Center in Z direction

60

List of Figures

11 The OctoMag prototype is designed and constructed at ETH ZurichThe system contains eight electromagnets in closed configuration [6] 6

12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltech-nologie GmbH Rostock-Warnemuende Germany) in three-dimensionalspace The magnetic system consists of 8 iron-core electromagnets thatsurrounds a water reservoir Magnetic-based robotic system give morefreedom in moving through trajectories compared to this system How-ever this system showed greater positioning accuracy compared to themagnet-based one [36] 7

21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console wherethe surgeon sits while operating a patient-side cart where the patient ispositioned during surgery interactive robotic arms a three-dimensionalhigh definition vision system and proprietary EndoWrist instruments[28] (a) The EndoWrist Insturment (b) Surgen console (c) visionsystem (d) Patient-side cart 13

22 ROBODOC Surgical system that enables pre-surgical 3D planningMeanwhile the ROBODOC Surgical assistant executes the pre-surgicalplan with unparalleled precision Benefits of the ROBODOC Systemover traditional methods include improved cavity fit fill and alignmentfor prosthetic implants [29] [30] [32] as well as a reduction of leg lengthdiscrepancy [30] and intra-operative fractures [29] [31] 14

23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom 15

61

List of Figures

24 Communication protocol architecture consisting of 5 layers It startswith the physical layer that deals with the physical (ie electricaland mechanical) aspects of transmitting data (eg voltage levels pin-connector design cable lengths and grounding arrangements) the Data-link layer deals with the transmission of data frames (eg packets) overa physical link between network entities including the incorporation oferror-correction coding into the data frames the network that dealswith establishing paths for data between a pair of computers and han-dling any switching among alternative routes between the computersas well as with definitions of how to break messages (or packets) upinto individual packets of data in such a way that the packets canbe transmitted and then reassembled Transport layer deals with datatransfer between end systems flow control for two computers Finallyapplication layer that deals with the interface between a user and thehost computer 16

25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero 17

26 Aimed position on a selected triangle in a Computer Aided Device file 19

27 Point calibration grid with orientation beam used to calibrate the cam-eras and extract the intrinsic parameters 20

28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is thestraight line which is perpendicular to the image plane and through theprojection center becomes optical axis CameraP point on the cameraframe ImageP point in the image frame wHc is the transformationfrom the world to camera frame f focal length C is the image startingpoint and it is the interaction between the optical axis and the imageplane 22

29 This image show the yz-view and the xz-view of the camera In theyz-plane The CPy point is mapped to PPv using a beamset 23

210 The point calibration grid with orientation beam is implemented toextract the intrinsic parameters of the camera [40] 26

62

List of Figures

211 Digital image processing steps image acquisition is the first step orprocess of the fundamental steps of digital image processing Imageacquisition could be as simple as being given an image that is alreadyin digital form image enhancement is among the simplest and mostappealing areas of digital image processing The idea behind enhance-ment techniques is to bring out detail that is obscured image restortionis an area that also deals with improving the appearance of an imagemorphological processing deals with tools for extracting image compo-nents that are useful in the representation and description of shapesegmentation procedures partition an image into its constituent partsor objects object recognition is the process that assigns a label suchas vehicle to an object based on its descriptors Finally representa-tion and description almost always follow the output of a segmentationstage which usually is raw pixel data constituting either the boundaryof a region or all the points in the region itself 28

212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques 29

213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm Apermanent magnet is attached to the end-effector and its magnetizationvector and z7 are collinear 30

214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using0T7 Position of the permanent magnet with respect to the end-effectoris defined using 7TM Position of the microparticle with respect to themagnet is defined using MTp 34

215 Closed-loop motion control of the position of a paramagnetic micropar-ticle using the robotic arm (KUKA iiwa r800) holding the permanentmagnet The particles are contained in water inside a glass tube andtheir position (P) is determined using a camera system and a fea-ture tracking software The control system positions the microparticles(based on the given reference position (Pref) by controlling the positionand orientation of the permenant magnet through the robotic arm 35

31 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluidThe system consist of a robotic arm and a cylindrical NdFeB magnetattached to its end-effector Paramagnetic microparticles with averagediameter of 100 microm are contained inside a glass tube 38

63

List of Figures

32 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluid(a) The system consist of a robotic arm with 7 degrees-of-freedom anda cylindrical NdFeB magnet attached to its end-effector Paramag-netic microparticles (PLAParticles-M-redF-plain from Micromod Par-tikeltechnologie GmbH Rostock-Warnemuende Germany) with aver-age diameter of 100 microm are contained inside a glass tube and posi-tioned in three-dimensional (3D) space using the field gradients (b)The configuration of the robotic arm (indicated using the joint spacecoordinates q) solely control the position of the microparticles in 3Dspace (c) The microparticle follows a path (blue trajectory) control-lably based on the configuration of the robotic arm P1 P2 and P3 arethe position vectors of the microparticles corresponding to the roboticconfigurations q1 q2 and q3 respectively 39

33 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the KUKA LBR iiwa 7R800 robotic arm The black arrow indicates the position of the clusterin the xy-plane whereas the blue line represents the reference posi-tion along z-axis (a) A cluster of microparticle is controlled to movedownward towards a reference position under the influence of controlledmagnetic field gradient and its own weight The magnetic field gradientis controlled via the configuration of the robotic arm (b) The clusteris pulled up towards the reference position using the field gradient 41

34 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the end-effector of theKUKA LBR iiwa 7 R800 robotic arm The microparticles is controlledto move downward at an average speed of 58 mms and is pulledupwards at speed of 71 mms 42

35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the roboticarm (a) Position of the microparticles along x-axis The microparticlesmove at average speed of 165 mms along x-axis (b) Motion of themicroparticles along y-axis towards the reference position at an averagespeed of 025 mms (c d e and f) Configuration of the robotic armduring the motion control of the microparticles is represented using qjfor j = 1 7 43

64

List of Figures

36 Kinematic control of a paramagnetic microparticle is achieved usinga permanent magnet attached to the end-effector of the robotic arm(a) A cluster of paramagnetic microparticles is controlled in three-dimensional space towards a reference position (vertical red line) Thecluster is pulled towards the reference position at an average speed of143 mms and 028 mms along x- and y-axis respectively (b) Po-sition of the end-effector of the robotic arm is calculated during theclosed-loop control of the microparticles (c d e and f) Configurationof the robotic arm during the control of the microparticle 44

37 Straight line and sinusoidal trajectory tracking by the controlled mi-croparticles via the controlled magnetic field gradient Waypoints (smallred circles) are defined along the reference trajectory (red lines) andcontrol law (232) is implemented based on (231) 45

38 The configuration of the robotic arm during trajectory tracking of asinusoidal reference trajectory is represented using qj for j = 1 7 46

39 Representative configurations for the robotic are during the tracking ofa sinusoidal trajectory 46

1 Sending message through User Datagram Protocol(UDP) Consists of amessage of 192 bytes that holds the information about the robot stateFirst 24 byts show the cartesian stiffness then 28 bytes that show theaxis stiffness followd by 24 bytes that contains the cartesian dampingthen another 28 bytes with the axis damping information Further 28bytes with the Joints positions values followed by 24 bytes with thecartesian positions Then 4 bytes for the position flag and the messageends with 32 bytes that shows the digital outputs state 70

2 Receiving a message through User Datagram Protocol(UDP) consist-ing of 192 bytes that hold the information about the robot state first28 bytes show the information of the joings angles the next 24 bytesthat show the cartesian position followd by 28 bytes that contains thetorques of the joints then another 28 bytes with the external torquesfurther 12 bytes with the external forces followed by 8 bytes with thetimestamps Then 32 bytes for the digital inputs and the message endswith 32 bytes that shows the digital outputs state 71

3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras 74

65

List of Tables

21 This table shows the definitions of the intrinsic parameters that areextracted 21

22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ithframe of reference 31

31 Setup specifications used in the experiments 40

67

Appendix

1 Communication protocol User Datagram Protocol (UDP)

In this thesis we use UDP as our mean of communication between C and iiwa server(uses java as programming language) the reason behind choosing UDP is because wewanted to transfer the data biderectionally as fast as possible With UDP we cantransfer data in the optimal scenario every 4 millisecond wheras in TCP data thefastest we can get out of the protocol is 8msAn example of how to create a UDPcommunication is shown in the appendix

1001 Sending bytes message from robot server to C

Every 4 ms the windows application which acts as a server in this case sends 192 bytesmessage to the robot server which takes the role of the client in this case The messageis divided into sections as shown in the Figure 1 An example is shown in the appendixto explain how to send byte messages from the robot server or from the c application

The first 24 bytes contains details about the Cartesian stiffness the first 4 bytes aredecoded afterward to extract the stiffness value in X the 4 follow bytes decoded tostiffness value in Y then the next 4 decoded to stiffness value in Z then stiffness valuein A stiffness value in B and finally the stiffness value in C The next 28 bytes holdthe details of the 7 axis stiffness each 4 bytes are decoded to extract the stiffness ofA1 A2 A3 A4 A5 A6 A7 respectively Followed with another 24 bytes which holddetails about Cartesian damping each 4 bytes are decoded to extract the float values

69

List of Tables

Figure 1 Sending message through User Datagram Protocol(UDP) Consists of a mes-sage of 192 bytes that holds the information about the robot state First 24byts show the cartesian stiffness then 28 bytes that show the axis stiffnessfollowd by 24 bytes that contains the cartesian damping then another 28bytes with the axis damping information Further 28 bytes with the Jointspositions values followed by 24 bytes with the cartesian positions Then 4bytes for the position flag and the message ends with 32 bytes that showsthe digital outputs state

of damping in X Y Z A B C respectively Followed with 28 bytes which hold detailsabout Axis damping each 4 bytes are decoded to extract the float values of dampingin A1 A2 A3 A4 A5 A6 A7 respectively Followed with another 28 bytes whichhold details about joint positions each 4 bytes are decoded to extract the float valuesof joint angle (in radian) in A1 A2 A3 A4 A5 A6 A7 respectively Followed with24 bytes which hold details about Cartesian positions each 4 bytes are decoded toextract the float values of XY and Z (in mm) followed by A B and C (in radian)respectively Followed with 4 bytes that is decoded to a float number between 1 and4 defined as follows

1 No movement2 Gravity compensation movement3 Axis movement4 Cartesian movement

Finally the last 32 bytes are used to transfer the values of digital outputs each 2 bytesare decoded to give an integer value either 0 or 1 for each output of the 16 outputs

0 Output of turned off1 Output of turned on

70

1 Communication protocol User Datagram Protocol (UDP)

1002 Receiving byte message from C application in the robot serverOn a different thread every 4 ms the Robot server which acts as a server in this casesends 192 bytes message to the windows application which takes the role of the clientin this case The message is divided into sections as shown in the Figure 2 An exam-ple of how to receive message in c and in the robot message is shown in the appendix

Figure 2 Receiving a message through User Datagram Protocol(UDP) consisting of192 bytes that hold the information about the robot state first 28 bytesshow the information of the joings angles the next 24 bytes that show thecartesian position followd by 28 bytes that contains the torques of the jointsthen another 28 bytes with the external torques further 12 bytes with theexternal forces followed by 8 bytes with the timestamps Then 32 bytes forthe digital inputs and the message ends with 32 bytes that shows the digitaloutputs state

The First 28 bytes shows the Joint Positions These bytes are decoded as explained inthe sending paragraph first 4 bytes have Joints position (in Radian) for A1 A2 A3A4 A5 A6 A7 respectively Followed by 24 bytes that shows the measured Cartesianposition X Y Z (in mm) then A B C (in Radian) respectively Followed by 28 bytesthat contains the measured axis torques (in Nm) for A1 A2 A3 A4 A5 A6 finallyA7 respectively Followed by another 28 bytes that shows the external axis torques(in Nm) for A1 A2 A3 A4 A5 A6 A7 respectively Then 12 bytes that showsthe external forces (in Newton) for Cartesian X Y and Z respectively Followed by 4bytes that is decoded to float that shows the measured torque timestamp (in seconds)Followed by another 4 bytes that is decoded also to another float that shows externaltorque timestamp (in seconds) The last two packets of data contain the details ofthe digital inputsoutputs the first 32 bytes are for 16 digital inputs each 2 bytes aredecoded into an integer of float that is either 0 or 10 Input of turn off1 Input of turn on

71

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

Bibliography

sθ Sine Theta is part of the DH parameter

stx1 Selected triangle vertices 1 in X direction

stx2 Selected triangle vertices 2 in X direction

stx3 Selected triangle vertices 3 in X direction

sty1 Selected triangle vertices 1 in y direction

sty2 Selected triangle vertices 2 in y direction

sty3 Selected triangle vertices 3 in y direction

stz1 Selected triangle vertices 1 in z direction

stz2 Selected triangle vertices 2 in z direction

stz3 Selected triangle vertices 3 in z direction

T Transformation matrix

v Volume of microparticle

Xm Magnetic susceptibility constant

B(P) Magnetic field with respect to particle position

DH Parameter denavit hartenberg parameter

e Position error

F(P) Magnetic Force with respect to the particle position

M Magnetization of the permanent magnet

59

Bibliography

TC Selected triangle Center

TCX Selected triangle Center in X direction

TCY Selected triangle Center in Y direction

TCZ Selected triangle Center in Z direction

60

List of Figures

11 The OctoMag prototype is designed and constructed at ETH ZurichThe system contains eight electromagnets in closed configuration [6] 6

12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltech-nologie GmbH Rostock-Warnemuende Germany) in three-dimensionalspace The magnetic system consists of 8 iron-core electromagnets thatsurrounds a water reservoir Magnetic-based robotic system give morefreedom in moving through trajectories compared to this system How-ever this system showed greater positioning accuracy compared to themagnet-based one [36] 7

21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console wherethe surgeon sits while operating a patient-side cart where the patient ispositioned during surgery interactive robotic arms a three-dimensionalhigh definition vision system and proprietary EndoWrist instruments[28] (a) The EndoWrist Insturment (b) Surgen console (c) visionsystem (d) Patient-side cart 13

22 ROBODOC Surgical system that enables pre-surgical 3D planningMeanwhile the ROBODOC Surgical assistant executes the pre-surgicalplan with unparalleled precision Benefits of the ROBODOC Systemover traditional methods include improved cavity fit fill and alignmentfor prosthetic implants [29] [30] [32] as well as a reduction of leg lengthdiscrepancy [30] and intra-operative fractures [29] [31] 14

23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom 15

61

List of Figures

24 Communication protocol architecture consisting of 5 layers It startswith the physical layer that deals with the physical (ie electricaland mechanical) aspects of transmitting data (eg voltage levels pin-connector design cable lengths and grounding arrangements) the Data-link layer deals with the transmission of data frames (eg packets) overa physical link between network entities including the incorporation oferror-correction coding into the data frames the network that dealswith establishing paths for data between a pair of computers and han-dling any switching among alternative routes between the computersas well as with definitions of how to break messages (or packets) upinto individual packets of data in such a way that the packets canbe transmitted and then reassembled Transport layer deals with datatransfer between end systems flow control for two computers Finallyapplication layer that deals with the interface between a user and thehost computer 16

25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero 17

26 Aimed position on a selected triangle in a Computer Aided Device file 19

27 Point calibration grid with orientation beam used to calibrate the cam-eras and extract the intrinsic parameters 20

28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is thestraight line which is perpendicular to the image plane and through theprojection center becomes optical axis CameraP point on the cameraframe ImageP point in the image frame wHc is the transformationfrom the world to camera frame f focal length C is the image startingpoint and it is the interaction between the optical axis and the imageplane 22

29 This image show the yz-view and the xz-view of the camera In theyz-plane The CPy point is mapped to PPv using a beamset 23

210 The point calibration grid with orientation beam is implemented toextract the intrinsic parameters of the camera [40] 26

62

List of Figures

211 Digital image processing steps image acquisition is the first step orprocess of the fundamental steps of digital image processing Imageacquisition could be as simple as being given an image that is alreadyin digital form image enhancement is among the simplest and mostappealing areas of digital image processing The idea behind enhance-ment techniques is to bring out detail that is obscured image restortionis an area that also deals with improving the appearance of an imagemorphological processing deals with tools for extracting image compo-nents that are useful in the representation and description of shapesegmentation procedures partition an image into its constituent partsor objects object recognition is the process that assigns a label suchas vehicle to an object based on its descriptors Finally representa-tion and description almost always follow the output of a segmentationstage which usually is raw pixel data constituting either the boundaryof a region or all the points in the region itself 28

212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques 29

213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm Apermanent magnet is attached to the end-effector and its magnetizationvector and z7 are collinear 30

214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using0T7 Position of the permanent magnet with respect to the end-effectoris defined using 7TM Position of the microparticle with respect to themagnet is defined using MTp 34

215 Closed-loop motion control of the position of a paramagnetic micropar-ticle using the robotic arm (KUKA iiwa r800) holding the permanentmagnet The particles are contained in water inside a glass tube andtheir position (P) is determined using a camera system and a fea-ture tracking software The control system positions the microparticles(based on the given reference position (Pref) by controlling the positionand orientation of the permenant magnet through the robotic arm 35

31 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluidThe system consist of a robotic arm and a cylindrical NdFeB magnetattached to its end-effector Paramagnetic microparticles with averagediameter of 100 microm are contained inside a glass tube 38

63

List of Figures

32 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluid(a) The system consist of a robotic arm with 7 degrees-of-freedom anda cylindrical NdFeB magnet attached to its end-effector Paramag-netic microparticles (PLAParticles-M-redF-plain from Micromod Par-tikeltechnologie GmbH Rostock-Warnemuende Germany) with aver-age diameter of 100 microm are contained inside a glass tube and posi-tioned in three-dimensional (3D) space using the field gradients (b)The configuration of the robotic arm (indicated using the joint spacecoordinates q) solely control the position of the microparticles in 3Dspace (c) The microparticle follows a path (blue trajectory) control-lably based on the configuration of the robotic arm P1 P2 and P3 arethe position vectors of the microparticles corresponding to the roboticconfigurations q1 q2 and q3 respectively 39

33 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the KUKA LBR iiwa 7R800 robotic arm The black arrow indicates the position of the clusterin the xy-plane whereas the blue line represents the reference posi-tion along z-axis (a) A cluster of microparticle is controlled to movedownward towards a reference position under the influence of controlledmagnetic field gradient and its own weight The magnetic field gradientis controlled via the configuration of the robotic arm (b) The clusteris pulled up towards the reference position using the field gradient 41

34 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the end-effector of theKUKA LBR iiwa 7 R800 robotic arm The microparticles is controlledto move downward at an average speed of 58 mms and is pulledupwards at speed of 71 mms 42

35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the roboticarm (a) Position of the microparticles along x-axis The microparticlesmove at average speed of 165 mms along x-axis (b) Motion of themicroparticles along y-axis towards the reference position at an averagespeed of 025 mms (c d e and f) Configuration of the robotic armduring the motion control of the microparticles is represented using qjfor j = 1 7 43

64

List of Figures

36 Kinematic control of a paramagnetic microparticle is achieved usinga permanent magnet attached to the end-effector of the robotic arm(a) A cluster of paramagnetic microparticles is controlled in three-dimensional space towards a reference position (vertical red line) Thecluster is pulled towards the reference position at an average speed of143 mms and 028 mms along x- and y-axis respectively (b) Po-sition of the end-effector of the robotic arm is calculated during theclosed-loop control of the microparticles (c d e and f) Configurationof the robotic arm during the control of the microparticle 44

37 Straight line and sinusoidal trajectory tracking by the controlled mi-croparticles via the controlled magnetic field gradient Waypoints (smallred circles) are defined along the reference trajectory (red lines) andcontrol law (232) is implemented based on (231) 45

38 The configuration of the robotic arm during trajectory tracking of asinusoidal reference trajectory is represented using qj for j = 1 7 46

39 Representative configurations for the robotic are during the tracking ofa sinusoidal trajectory 46

1 Sending message through User Datagram Protocol(UDP) Consists of amessage of 192 bytes that holds the information about the robot stateFirst 24 byts show the cartesian stiffness then 28 bytes that show theaxis stiffness followd by 24 bytes that contains the cartesian dampingthen another 28 bytes with the axis damping information Further 28bytes with the Joints positions values followed by 24 bytes with thecartesian positions Then 4 bytes for the position flag and the messageends with 32 bytes that shows the digital outputs state 70

2 Receiving a message through User Datagram Protocol(UDP) consist-ing of 192 bytes that hold the information about the robot state first28 bytes show the information of the joings angles the next 24 bytesthat show the cartesian position followd by 28 bytes that contains thetorques of the joints then another 28 bytes with the external torquesfurther 12 bytes with the external forces followed by 8 bytes with thetimestamps Then 32 bytes for the digital inputs and the message endswith 32 bytes that shows the digital outputs state 71

3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras 74

65

List of Tables

21 This table shows the definitions of the intrinsic parameters that areextracted 21

22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ithframe of reference 31

31 Setup specifications used in the experiments 40

67

Appendix

1 Communication protocol User Datagram Protocol (UDP)

In this thesis we use UDP as our mean of communication between C and iiwa server(uses java as programming language) the reason behind choosing UDP is because wewanted to transfer the data biderectionally as fast as possible With UDP we cantransfer data in the optimal scenario every 4 millisecond wheras in TCP data thefastest we can get out of the protocol is 8msAn example of how to create a UDPcommunication is shown in the appendix

1001 Sending bytes message from robot server to C

Every 4 ms the windows application which acts as a server in this case sends 192 bytesmessage to the robot server which takes the role of the client in this case The messageis divided into sections as shown in the Figure 1 An example is shown in the appendixto explain how to send byte messages from the robot server or from the c application

The first 24 bytes contains details about the Cartesian stiffness the first 4 bytes aredecoded afterward to extract the stiffness value in X the 4 follow bytes decoded tostiffness value in Y then the next 4 decoded to stiffness value in Z then stiffness valuein A stiffness value in B and finally the stiffness value in C The next 28 bytes holdthe details of the 7 axis stiffness each 4 bytes are decoded to extract the stiffness ofA1 A2 A3 A4 A5 A6 A7 respectively Followed with another 24 bytes which holddetails about Cartesian damping each 4 bytes are decoded to extract the float values

69

List of Tables

Figure 1 Sending message through User Datagram Protocol(UDP) Consists of a mes-sage of 192 bytes that holds the information about the robot state First 24byts show the cartesian stiffness then 28 bytes that show the axis stiffnessfollowd by 24 bytes that contains the cartesian damping then another 28bytes with the axis damping information Further 28 bytes with the Jointspositions values followed by 24 bytes with the cartesian positions Then 4bytes for the position flag and the message ends with 32 bytes that showsthe digital outputs state

of damping in X Y Z A B C respectively Followed with 28 bytes which hold detailsabout Axis damping each 4 bytes are decoded to extract the float values of dampingin A1 A2 A3 A4 A5 A6 A7 respectively Followed with another 28 bytes whichhold details about joint positions each 4 bytes are decoded to extract the float valuesof joint angle (in radian) in A1 A2 A3 A4 A5 A6 A7 respectively Followed with24 bytes which hold details about Cartesian positions each 4 bytes are decoded toextract the float values of XY and Z (in mm) followed by A B and C (in radian)respectively Followed with 4 bytes that is decoded to a float number between 1 and4 defined as follows

1 No movement2 Gravity compensation movement3 Axis movement4 Cartesian movement

Finally the last 32 bytes are used to transfer the values of digital outputs each 2 bytesare decoded to give an integer value either 0 or 1 for each output of the 16 outputs

0 Output of turned off1 Output of turned on

70

1 Communication protocol User Datagram Protocol (UDP)

1002 Receiving byte message from C application in the robot serverOn a different thread every 4 ms the Robot server which acts as a server in this casesends 192 bytes message to the windows application which takes the role of the clientin this case The message is divided into sections as shown in the Figure 2 An exam-ple of how to receive message in c and in the robot message is shown in the appendix

Figure 2 Receiving a message through User Datagram Protocol(UDP) consisting of192 bytes that hold the information about the robot state first 28 bytesshow the information of the joings angles the next 24 bytes that show thecartesian position followd by 28 bytes that contains the torques of the jointsthen another 28 bytes with the external torques further 12 bytes with theexternal forces followed by 8 bytes with the timestamps Then 32 bytes forthe digital inputs and the message ends with 32 bytes that shows the digitaloutputs state

The First 28 bytes shows the Joint Positions These bytes are decoded as explained inthe sending paragraph first 4 bytes have Joints position (in Radian) for A1 A2 A3A4 A5 A6 A7 respectively Followed by 24 bytes that shows the measured Cartesianposition X Y Z (in mm) then A B C (in Radian) respectively Followed by 28 bytesthat contains the measured axis torques (in Nm) for A1 A2 A3 A4 A5 A6 finallyA7 respectively Followed by another 28 bytes that shows the external axis torques(in Nm) for A1 A2 A3 A4 A5 A6 A7 respectively Then 12 bytes that showsthe external forces (in Newton) for Cartesian X Y and Z respectively Followed by 4bytes that is decoded to float that shows the measured torque timestamp (in seconds)Followed by another 4 bytes that is decoded also to another float that shows externaltorque timestamp (in seconds) The last two packets of data contain the details ofthe digital inputsoutputs the first 32 bytes are for 16 digital inputs each 2 bytes aredecoded into an integer of float that is either 0 or 10 Input of turn off1 Input of turn on

71

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

Bibliography

TC Selected triangle Center

TCX Selected triangle Center in X direction

TCY Selected triangle Center in Y direction

TCZ Selected triangle Center in Z direction

60

List of Figures

11 The OctoMag prototype is designed and constructed at ETH ZurichThe system contains eight electromagnets in closed configuration [6] 6

12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltech-nologie GmbH Rostock-Warnemuende Germany) in three-dimensionalspace The magnetic system consists of 8 iron-core electromagnets thatsurrounds a water reservoir Magnetic-based robotic system give morefreedom in moving through trajectories compared to this system How-ever this system showed greater positioning accuracy compared to themagnet-based one [36] 7

21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console wherethe surgeon sits while operating a patient-side cart where the patient ispositioned during surgery interactive robotic arms a three-dimensionalhigh definition vision system and proprietary EndoWrist instruments[28] (a) The EndoWrist Insturment (b) Surgen console (c) visionsystem (d) Patient-side cart 13

22 ROBODOC Surgical system that enables pre-surgical 3D planningMeanwhile the ROBODOC Surgical assistant executes the pre-surgicalplan with unparalleled precision Benefits of the ROBODOC Systemover traditional methods include improved cavity fit fill and alignmentfor prosthetic implants [29] [30] [32] as well as a reduction of leg lengthdiscrepancy [30] and intra-operative fractures [29] [31] 14

23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom 15

61

List of Figures

24 Communication protocol architecture consisting of 5 layers It startswith the physical layer that deals with the physical (ie electricaland mechanical) aspects of transmitting data (eg voltage levels pin-connector design cable lengths and grounding arrangements) the Data-link layer deals with the transmission of data frames (eg packets) overa physical link between network entities including the incorporation oferror-correction coding into the data frames the network that dealswith establishing paths for data between a pair of computers and han-dling any switching among alternative routes between the computersas well as with definitions of how to break messages (or packets) upinto individual packets of data in such a way that the packets canbe transmitted and then reassembled Transport layer deals with datatransfer between end systems flow control for two computers Finallyapplication layer that deals with the interface between a user and thehost computer 16

25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero 17

26 Aimed position on a selected triangle in a Computer Aided Device file 19

27 Point calibration grid with orientation beam used to calibrate the cam-eras and extract the intrinsic parameters 20

28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is thestraight line which is perpendicular to the image plane and through theprojection center becomes optical axis CameraP point on the cameraframe ImageP point in the image frame wHc is the transformationfrom the world to camera frame f focal length C is the image startingpoint and it is the interaction between the optical axis and the imageplane 22

29 This image show the yz-view and the xz-view of the camera In theyz-plane The CPy point is mapped to PPv using a beamset 23

210 The point calibration grid with orientation beam is implemented toextract the intrinsic parameters of the camera [40] 26

62

List of Figures

211 Digital image processing steps image acquisition is the first step orprocess of the fundamental steps of digital image processing Imageacquisition could be as simple as being given an image that is alreadyin digital form image enhancement is among the simplest and mostappealing areas of digital image processing The idea behind enhance-ment techniques is to bring out detail that is obscured image restortionis an area that also deals with improving the appearance of an imagemorphological processing deals with tools for extracting image compo-nents that are useful in the representation and description of shapesegmentation procedures partition an image into its constituent partsor objects object recognition is the process that assigns a label suchas vehicle to an object based on its descriptors Finally representa-tion and description almost always follow the output of a segmentationstage which usually is raw pixel data constituting either the boundaryof a region or all the points in the region itself 28

212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques 29

213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm Apermanent magnet is attached to the end-effector and its magnetizationvector and z7 are collinear 30

214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using0T7 Position of the permanent magnet with respect to the end-effectoris defined using 7TM Position of the microparticle with respect to themagnet is defined using MTp 34

215 Closed-loop motion control of the position of a paramagnetic micropar-ticle using the robotic arm (KUKA iiwa r800) holding the permanentmagnet The particles are contained in water inside a glass tube andtheir position (P) is determined using a camera system and a fea-ture tracking software The control system positions the microparticles(based on the given reference position (Pref) by controlling the positionand orientation of the permenant magnet through the robotic arm 35

31 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluidThe system consist of a robotic arm and a cylindrical NdFeB magnetattached to its end-effector Paramagnetic microparticles with averagediameter of 100 microm are contained inside a glass tube 38

63

List of Figures

32 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluid(a) The system consist of a robotic arm with 7 degrees-of-freedom anda cylindrical NdFeB magnet attached to its end-effector Paramag-netic microparticles (PLAParticles-M-redF-plain from Micromod Par-tikeltechnologie GmbH Rostock-Warnemuende Germany) with aver-age diameter of 100 microm are contained inside a glass tube and posi-tioned in three-dimensional (3D) space using the field gradients (b)The configuration of the robotic arm (indicated using the joint spacecoordinates q) solely control the position of the microparticles in 3Dspace (c) The microparticle follows a path (blue trajectory) control-lably based on the configuration of the robotic arm P1 P2 and P3 arethe position vectors of the microparticles corresponding to the roboticconfigurations q1 q2 and q3 respectively 39

33 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the KUKA LBR iiwa 7R800 robotic arm The black arrow indicates the position of the clusterin the xy-plane whereas the blue line represents the reference posi-tion along z-axis (a) A cluster of microparticle is controlled to movedownward towards a reference position under the influence of controlledmagnetic field gradient and its own weight The magnetic field gradientis controlled via the configuration of the robotic arm (b) The clusteris pulled up towards the reference position using the field gradient 41

34 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the end-effector of theKUKA LBR iiwa 7 R800 robotic arm The microparticles is controlledto move downward at an average speed of 58 mms and is pulledupwards at speed of 71 mms 42

35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the roboticarm (a) Position of the microparticles along x-axis The microparticlesmove at average speed of 165 mms along x-axis (b) Motion of themicroparticles along y-axis towards the reference position at an averagespeed of 025 mms (c d e and f) Configuration of the robotic armduring the motion control of the microparticles is represented using qjfor j = 1 7 43

64

List of Figures

36 Kinematic control of a paramagnetic microparticle is achieved usinga permanent magnet attached to the end-effector of the robotic arm(a) A cluster of paramagnetic microparticles is controlled in three-dimensional space towards a reference position (vertical red line) Thecluster is pulled towards the reference position at an average speed of143 mms and 028 mms along x- and y-axis respectively (b) Po-sition of the end-effector of the robotic arm is calculated during theclosed-loop control of the microparticles (c d e and f) Configurationof the robotic arm during the control of the microparticle 44

37 Straight line and sinusoidal trajectory tracking by the controlled mi-croparticles via the controlled magnetic field gradient Waypoints (smallred circles) are defined along the reference trajectory (red lines) andcontrol law (232) is implemented based on (231) 45

38 The configuration of the robotic arm during trajectory tracking of asinusoidal reference trajectory is represented using qj for j = 1 7 46

39 Representative configurations for the robotic are during the tracking ofa sinusoidal trajectory 46

1 Sending message through User Datagram Protocol(UDP) Consists of amessage of 192 bytes that holds the information about the robot stateFirst 24 byts show the cartesian stiffness then 28 bytes that show theaxis stiffness followd by 24 bytes that contains the cartesian dampingthen another 28 bytes with the axis damping information Further 28bytes with the Joints positions values followed by 24 bytes with thecartesian positions Then 4 bytes for the position flag and the messageends with 32 bytes that shows the digital outputs state 70

2 Receiving a message through User Datagram Protocol(UDP) consist-ing of 192 bytes that hold the information about the robot state first28 bytes show the information of the joings angles the next 24 bytesthat show the cartesian position followd by 28 bytes that contains thetorques of the joints then another 28 bytes with the external torquesfurther 12 bytes with the external forces followed by 8 bytes with thetimestamps Then 32 bytes for the digital inputs and the message endswith 32 bytes that shows the digital outputs state 71

3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras 74

65

List of Tables

21 This table shows the definitions of the intrinsic parameters that areextracted 21

22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ithframe of reference 31

31 Setup specifications used in the experiments 40

67

Appendix

1 Communication protocol User Datagram Protocol (UDP)

In this thesis we use UDP as our mean of communication between C and iiwa server(uses java as programming language) the reason behind choosing UDP is because wewanted to transfer the data biderectionally as fast as possible With UDP we cantransfer data in the optimal scenario every 4 millisecond wheras in TCP data thefastest we can get out of the protocol is 8msAn example of how to create a UDPcommunication is shown in the appendix

1001 Sending bytes message from robot server to C

Every 4 ms the windows application which acts as a server in this case sends 192 bytesmessage to the robot server which takes the role of the client in this case The messageis divided into sections as shown in the Figure 1 An example is shown in the appendixto explain how to send byte messages from the robot server or from the c application

The first 24 bytes contains details about the Cartesian stiffness the first 4 bytes aredecoded afterward to extract the stiffness value in X the 4 follow bytes decoded tostiffness value in Y then the next 4 decoded to stiffness value in Z then stiffness valuein A stiffness value in B and finally the stiffness value in C The next 28 bytes holdthe details of the 7 axis stiffness each 4 bytes are decoded to extract the stiffness ofA1 A2 A3 A4 A5 A6 A7 respectively Followed with another 24 bytes which holddetails about Cartesian damping each 4 bytes are decoded to extract the float values

69

List of Tables

Figure 1 Sending message through User Datagram Protocol(UDP) Consists of a mes-sage of 192 bytes that holds the information about the robot state First 24byts show the cartesian stiffness then 28 bytes that show the axis stiffnessfollowd by 24 bytes that contains the cartesian damping then another 28bytes with the axis damping information Further 28 bytes with the Jointspositions values followed by 24 bytes with the cartesian positions Then 4bytes for the position flag and the message ends with 32 bytes that showsthe digital outputs state

of damping in X Y Z A B C respectively Followed with 28 bytes which hold detailsabout Axis damping each 4 bytes are decoded to extract the float values of dampingin A1 A2 A3 A4 A5 A6 A7 respectively Followed with another 28 bytes whichhold details about joint positions each 4 bytes are decoded to extract the float valuesof joint angle (in radian) in A1 A2 A3 A4 A5 A6 A7 respectively Followed with24 bytes which hold details about Cartesian positions each 4 bytes are decoded toextract the float values of XY and Z (in mm) followed by A B and C (in radian)respectively Followed with 4 bytes that is decoded to a float number between 1 and4 defined as follows

1 No movement2 Gravity compensation movement3 Axis movement4 Cartesian movement

Finally the last 32 bytes are used to transfer the values of digital outputs each 2 bytesare decoded to give an integer value either 0 or 1 for each output of the 16 outputs

0 Output of turned off1 Output of turned on

70

1 Communication protocol User Datagram Protocol (UDP)

1002 Receiving byte message from C application in the robot serverOn a different thread every 4 ms the Robot server which acts as a server in this casesends 192 bytes message to the windows application which takes the role of the clientin this case The message is divided into sections as shown in the Figure 2 An exam-ple of how to receive message in c and in the robot message is shown in the appendix

Figure 2 Receiving a message through User Datagram Protocol(UDP) consisting of192 bytes that hold the information about the robot state first 28 bytesshow the information of the joings angles the next 24 bytes that show thecartesian position followd by 28 bytes that contains the torques of the jointsthen another 28 bytes with the external torques further 12 bytes with theexternal forces followed by 8 bytes with the timestamps Then 32 bytes forthe digital inputs and the message ends with 32 bytes that shows the digitaloutputs state

The First 28 bytes shows the Joint Positions These bytes are decoded as explained inthe sending paragraph first 4 bytes have Joints position (in Radian) for A1 A2 A3A4 A5 A6 A7 respectively Followed by 24 bytes that shows the measured Cartesianposition X Y Z (in mm) then A B C (in Radian) respectively Followed by 28 bytesthat contains the measured axis torques (in Nm) for A1 A2 A3 A4 A5 A6 finallyA7 respectively Followed by another 28 bytes that shows the external axis torques(in Nm) for A1 A2 A3 A4 A5 A6 A7 respectively Then 12 bytes that showsthe external forces (in Newton) for Cartesian X Y and Z respectively Followed by 4bytes that is decoded to float that shows the measured torque timestamp (in seconds)Followed by another 4 bytes that is decoded also to another float that shows externaltorque timestamp (in seconds) The last two packets of data contain the details ofthe digital inputsoutputs the first 32 bytes are for 16 digital inputs each 2 bytes aredecoded into an integer of float that is either 0 or 10 Input of turn off1 Input of turn on

71

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

List of Figures

11 The OctoMag prototype is designed and constructed at ETH ZurichThe system contains eight electromagnets in closed configuration [6] 6

12 Magnetic system for the closed-loop motion control of paramagnetic mi-croparticles (PLAParticles-M-redF-plain from Micromod Partikeltech-nologie GmbH Rostock-Warnemuende Germany) in three-dimensionalspace The magnetic system consists of 8 iron-core electromagnets thatsurrounds a water reservoir Magnetic-based robotic system give morefreedom in moving through trajectories compared to this system How-ever this system showed greater positioning accuracy compared to themagnet-based one [36] 7

21 Da vinci robot surgical system (Intuitive surgical company SunnyvaleCalifornia) enables surgeons to perform delicate and complex operationsthrough a few small incisions The da Vinci System consists of severalkey components including an ergonomically designed console wherethe surgeon sits while operating a patient-side cart where the patient ispositioned during surgery interactive robotic arms a three-dimensionalhigh definition vision system and proprietary EndoWrist instruments[28] (a) The EndoWrist Insturment (b) Surgen console (c) visionsystem (d) Patient-side cart 13

22 ROBODOC Surgical system that enables pre-surgical 3D planningMeanwhile the ROBODOC Surgical assistant executes the pre-surgicalplan with unparalleled precision Benefits of the ROBODOC Systemover traditional methods include improved cavity fit fill and alignmentfor prosthetic implants [29] [30] [32] as well as a reduction of leg lengthdiscrepancy [30] and intra-operative fractures [29] [31] 14

23 The system consist of a robotic arm KUKA iiwa r800 with 7 degrees offreedom 15

61

List of Figures

24 Communication protocol architecture consisting of 5 layers It startswith the physical layer that deals with the physical (ie electricaland mechanical) aspects of transmitting data (eg voltage levels pin-connector design cable lengths and grounding arrangements) the Data-link layer deals with the transmission of data frames (eg packets) overa physical link between network entities including the incorporation oferror-correction coding into the data frames the network that dealswith establishing paths for data between a pair of computers and han-dling any switching among alternative routes between the computersas well as with definitions of how to break messages (or packets) upinto individual packets of data in such a way that the packets canbe transmitted and then reassembled Transport layer deals with datatransfer between end systems flow control for two computers Finallyapplication layer that deals with the interface between a user and thehost computer 16

25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero 17

26 Aimed position on a selected triangle in a Computer Aided Device file 19

27 Point calibration grid with orientation beam used to calibrate the cam-eras and extract the intrinsic parameters 20

28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is thestraight line which is perpendicular to the image plane and through theprojection center becomes optical axis CameraP point on the cameraframe ImageP point in the image frame wHc is the transformationfrom the world to camera frame f focal length C is the image startingpoint and it is the interaction between the optical axis and the imageplane 22

29 This image show the yz-view and the xz-view of the camera In theyz-plane The CPy point is mapped to PPv using a beamset 23

210 The point calibration grid with orientation beam is implemented toextract the intrinsic parameters of the camera [40] 26

62

List of Figures

211 Digital image processing steps image acquisition is the first step orprocess of the fundamental steps of digital image processing Imageacquisition could be as simple as being given an image that is alreadyin digital form image enhancement is among the simplest and mostappealing areas of digital image processing The idea behind enhance-ment techniques is to bring out detail that is obscured image restortionis an area that also deals with improving the appearance of an imagemorphological processing deals with tools for extracting image compo-nents that are useful in the representation and description of shapesegmentation procedures partition an image into its constituent partsor objects object recognition is the process that assigns a label suchas vehicle to an object based on its descriptors Finally representa-tion and description almost always follow the output of a segmentationstage which usually is raw pixel data constituting either the boundaryof a region or all the points in the region itself 28

212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques 29

213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm Apermanent magnet is attached to the end-effector and its magnetizationvector and z7 are collinear 30

214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using0T7 Position of the permanent magnet with respect to the end-effectoris defined using 7TM Position of the microparticle with respect to themagnet is defined using MTp 34

215 Closed-loop motion control of the position of a paramagnetic micropar-ticle using the robotic arm (KUKA iiwa r800) holding the permanentmagnet The particles are contained in water inside a glass tube andtheir position (P) is determined using a camera system and a fea-ture tracking software The control system positions the microparticles(based on the given reference position (Pref) by controlling the positionand orientation of the permenant magnet through the robotic arm 35

31 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluidThe system consist of a robotic arm and a cylindrical NdFeB magnetattached to its end-effector Paramagnetic microparticles with averagediameter of 100 microm are contained inside a glass tube 38

63

List of Figures

32 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluid(a) The system consist of a robotic arm with 7 degrees-of-freedom anda cylindrical NdFeB magnet attached to its end-effector Paramag-netic microparticles (PLAParticles-M-redF-plain from Micromod Par-tikeltechnologie GmbH Rostock-Warnemuende Germany) with aver-age diameter of 100 microm are contained inside a glass tube and posi-tioned in three-dimensional (3D) space using the field gradients (b)The configuration of the robotic arm (indicated using the joint spacecoordinates q) solely control the position of the microparticles in 3Dspace (c) The microparticle follows a path (blue trajectory) control-lably based on the configuration of the robotic arm P1 P2 and P3 arethe position vectors of the microparticles corresponding to the roboticconfigurations q1 q2 and q3 respectively 39

33 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the KUKA LBR iiwa 7R800 robotic arm The black arrow indicates the position of the clusterin the xy-plane whereas the blue line represents the reference posi-tion along z-axis (a) A cluster of microparticle is controlled to movedownward towards a reference position under the influence of controlledmagnetic field gradient and its own weight The magnetic field gradientis controlled via the configuration of the robotic arm (b) The clusteris pulled up towards the reference position using the field gradient 41

34 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the end-effector of theKUKA LBR iiwa 7 R800 robotic arm The microparticles is controlledto move downward at an average speed of 58 mms and is pulledupwards at speed of 71 mms 42

35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the roboticarm (a) Position of the microparticles along x-axis The microparticlesmove at average speed of 165 mms along x-axis (b) Motion of themicroparticles along y-axis towards the reference position at an averagespeed of 025 mms (c d e and f) Configuration of the robotic armduring the motion control of the microparticles is represented using qjfor j = 1 7 43

64

List of Figures

36 Kinematic control of a paramagnetic microparticle is achieved usinga permanent magnet attached to the end-effector of the robotic arm(a) A cluster of paramagnetic microparticles is controlled in three-dimensional space towards a reference position (vertical red line) Thecluster is pulled towards the reference position at an average speed of143 mms and 028 mms along x- and y-axis respectively (b) Po-sition of the end-effector of the robotic arm is calculated during theclosed-loop control of the microparticles (c d e and f) Configurationof the robotic arm during the control of the microparticle 44

37 Straight line and sinusoidal trajectory tracking by the controlled mi-croparticles via the controlled magnetic field gradient Waypoints (smallred circles) are defined along the reference trajectory (red lines) andcontrol law (232) is implemented based on (231) 45

38 The configuration of the robotic arm during trajectory tracking of asinusoidal reference trajectory is represented using qj for j = 1 7 46

39 Representative configurations for the robotic are during the tracking ofa sinusoidal trajectory 46

1 Sending message through User Datagram Protocol(UDP) Consists of amessage of 192 bytes that holds the information about the robot stateFirst 24 byts show the cartesian stiffness then 28 bytes that show theaxis stiffness followd by 24 bytes that contains the cartesian dampingthen another 28 bytes with the axis damping information Further 28bytes with the Joints positions values followed by 24 bytes with thecartesian positions Then 4 bytes for the position flag and the messageends with 32 bytes that shows the digital outputs state 70

2 Receiving a message through User Datagram Protocol(UDP) consist-ing of 192 bytes that hold the information about the robot state first28 bytes show the information of the joings angles the next 24 bytesthat show the cartesian position followd by 28 bytes that contains thetorques of the joints then another 28 bytes with the external torquesfurther 12 bytes with the external forces followed by 8 bytes with thetimestamps Then 32 bytes for the digital inputs and the message endswith 32 bytes that shows the digital outputs state 71

3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras 74

65

List of Tables

21 This table shows the definitions of the intrinsic parameters that areextracted 21

22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ithframe of reference 31

31 Setup specifications used in the experiments 40

67

Appendix

1 Communication protocol User Datagram Protocol (UDP)

In this thesis we use UDP as our mean of communication between C and iiwa server(uses java as programming language) the reason behind choosing UDP is because wewanted to transfer the data biderectionally as fast as possible With UDP we cantransfer data in the optimal scenario every 4 millisecond wheras in TCP data thefastest we can get out of the protocol is 8msAn example of how to create a UDPcommunication is shown in the appendix

1001 Sending bytes message from robot server to C

Every 4 ms the windows application which acts as a server in this case sends 192 bytesmessage to the robot server which takes the role of the client in this case The messageis divided into sections as shown in the Figure 1 An example is shown in the appendixto explain how to send byte messages from the robot server or from the c application

The first 24 bytes contains details about the Cartesian stiffness the first 4 bytes aredecoded afterward to extract the stiffness value in X the 4 follow bytes decoded tostiffness value in Y then the next 4 decoded to stiffness value in Z then stiffness valuein A stiffness value in B and finally the stiffness value in C The next 28 bytes holdthe details of the 7 axis stiffness each 4 bytes are decoded to extract the stiffness ofA1 A2 A3 A4 A5 A6 A7 respectively Followed with another 24 bytes which holddetails about Cartesian damping each 4 bytes are decoded to extract the float values

69

List of Tables

Figure 1 Sending message through User Datagram Protocol(UDP) Consists of a mes-sage of 192 bytes that holds the information about the robot state First 24byts show the cartesian stiffness then 28 bytes that show the axis stiffnessfollowd by 24 bytes that contains the cartesian damping then another 28bytes with the axis damping information Further 28 bytes with the Jointspositions values followed by 24 bytes with the cartesian positions Then 4bytes for the position flag and the message ends with 32 bytes that showsthe digital outputs state

of damping in X Y Z A B C respectively Followed with 28 bytes which hold detailsabout Axis damping each 4 bytes are decoded to extract the float values of dampingin A1 A2 A3 A4 A5 A6 A7 respectively Followed with another 28 bytes whichhold details about joint positions each 4 bytes are decoded to extract the float valuesof joint angle (in radian) in A1 A2 A3 A4 A5 A6 A7 respectively Followed with24 bytes which hold details about Cartesian positions each 4 bytes are decoded toextract the float values of XY and Z (in mm) followed by A B and C (in radian)respectively Followed with 4 bytes that is decoded to a float number between 1 and4 defined as follows

1 No movement2 Gravity compensation movement3 Axis movement4 Cartesian movement

Finally the last 32 bytes are used to transfer the values of digital outputs each 2 bytesare decoded to give an integer value either 0 or 1 for each output of the 16 outputs

0 Output of turned off1 Output of turned on

70

1 Communication protocol User Datagram Protocol (UDP)

1002 Receiving byte message from C application in the robot serverOn a different thread every 4 ms the Robot server which acts as a server in this casesends 192 bytes message to the windows application which takes the role of the clientin this case The message is divided into sections as shown in the Figure 2 An exam-ple of how to receive message in c and in the robot message is shown in the appendix

Figure 2 Receiving a message through User Datagram Protocol(UDP) consisting of192 bytes that hold the information about the robot state first 28 bytesshow the information of the joings angles the next 24 bytes that show thecartesian position followd by 28 bytes that contains the torques of the jointsthen another 28 bytes with the external torques further 12 bytes with theexternal forces followed by 8 bytes with the timestamps Then 32 bytes forthe digital inputs and the message ends with 32 bytes that shows the digitaloutputs state

The First 28 bytes shows the Joint Positions These bytes are decoded as explained inthe sending paragraph first 4 bytes have Joints position (in Radian) for A1 A2 A3A4 A5 A6 A7 respectively Followed by 24 bytes that shows the measured Cartesianposition X Y Z (in mm) then A B C (in Radian) respectively Followed by 28 bytesthat contains the measured axis torques (in Nm) for A1 A2 A3 A4 A5 A6 finallyA7 respectively Followed by another 28 bytes that shows the external axis torques(in Nm) for A1 A2 A3 A4 A5 A6 A7 respectively Then 12 bytes that showsthe external forces (in Newton) for Cartesian X Y and Z respectively Followed by 4bytes that is decoded to float that shows the measured torque timestamp (in seconds)Followed by another 4 bytes that is decoded also to another float that shows externaltorque timestamp (in seconds) The last two packets of data contain the details ofthe digital inputsoutputs the first 32 bytes are for 16 digital inputs each 2 bytes aredecoded into an integer of float that is either 0 or 10 Input of turn off1 Input of turn on

71

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

List of Figures

24 Communication protocol architecture consisting of 5 layers It startswith the physical layer that deals with the physical (ie electricaland mechanical) aspects of transmitting data (eg voltage levels pin-connector design cable lengths and grounding arrangements) the Data-link layer deals with the transmission of data frames (eg packets) overa physical link between network entities including the incorporation oferror-correction coding into the data frames the network that dealswith establishing paths for data between a pair of computers and han-dling any switching among alternative routes between the computersas well as with definitions of how to break messages (or packets) upinto individual packets of data in such a way that the packets canbe transmitted and then reassembled Transport layer deals with datatransfer between end systems flow control for two computers Finallyapplication layer that deals with the interface between a user and thehost computer 16

25 this picture shows gravity compensation button (The blue button) at-tached to the handler this button in connected to input 8 of the robotinputoutput and when pressed the catesian stiffness are set to zero 17

26 Aimed position on a selected triangle in a Computer Aided Device file 19

27 Point calibration grid with orientation beam used to calibrate the cam-eras and extract the intrinsic parameters 20

28 this figure shows the modelling of the camera the transformation fromworld frame to camera frame then to the image frame OV is thestraight line which is perpendicular to the image plane and through theprojection center becomes optical axis CameraP point on the cameraframe ImageP point in the image frame wHc is the transformationfrom the world to camera frame f focal length C is the image startingpoint and it is the interaction between the optical axis and the imageplane 22

29 This image show the yz-view and the xz-view of the camera In theyz-plane The CPy point is mapped to PPv using a beamset 23

210 The point calibration grid with orientation beam is implemented toextract the intrinsic parameters of the camera [40] 26

62

List of Figures

211 Digital image processing steps image acquisition is the first step orprocess of the fundamental steps of digital image processing Imageacquisition could be as simple as being given an image that is alreadyin digital form image enhancement is among the simplest and mostappealing areas of digital image processing The idea behind enhance-ment techniques is to bring out detail that is obscured image restortionis an area that also deals with improving the appearance of an imagemorphological processing deals with tools for extracting image compo-nents that are useful in the representation and description of shapesegmentation procedures partition an image into its constituent partsor objects object recognition is the process that assigns a label suchas vehicle to an object based on its descriptors Finally representa-tion and description almost always follow the output of a segmentationstage which usually is raw pixel data constituting either the boundaryof a region or all the points in the region itself 28

212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques 29

213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm Apermanent magnet is attached to the end-effector and its magnetizationvector and z7 are collinear 30

214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using0T7 Position of the permanent magnet with respect to the end-effectoris defined using 7TM Position of the microparticle with respect to themagnet is defined using MTp 34

215 Closed-loop motion control of the position of a paramagnetic micropar-ticle using the robotic arm (KUKA iiwa r800) holding the permanentmagnet The particles are contained in water inside a glass tube andtheir position (P) is determined using a camera system and a fea-ture tracking software The control system positions the microparticles(based on the given reference position (Pref) by controlling the positionand orientation of the permenant magnet through the robotic arm 35

31 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluidThe system consist of a robotic arm and a cylindrical NdFeB magnetattached to its end-effector Paramagnetic microparticles with averagediameter of 100 microm are contained inside a glass tube 38

63

List of Figures

32 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluid(a) The system consist of a robotic arm with 7 degrees-of-freedom anda cylindrical NdFeB magnet attached to its end-effector Paramag-netic microparticles (PLAParticles-M-redF-plain from Micromod Par-tikeltechnologie GmbH Rostock-Warnemuende Germany) with aver-age diameter of 100 microm are contained inside a glass tube and posi-tioned in three-dimensional (3D) space using the field gradients (b)The configuration of the robotic arm (indicated using the joint spacecoordinates q) solely control the position of the microparticles in 3Dspace (c) The microparticle follows a path (blue trajectory) control-lably based on the configuration of the robotic arm P1 P2 and P3 arethe position vectors of the microparticles corresponding to the roboticconfigurations q1 q2 and q3 respectively 39

33 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the KUKA LBR iiwa 7R800 robotic arm The black arrow indicates the position of the clusterin the xy-plane whereas the blue line represents the reference posi-tion along z-axis (a) A cluster of microparticle is controlled to movedownward towards a reference position under the influence of controlledmagnetic field gradient and its own weight The magnetic field gradientis controlled via the configuration of the robotic arm (b) The clusteris pulled up towards the reference position using the field gradient 41

34 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the end-effector of theKUKA LBR iiwa 7 R800 robotic arm The microparticles is controlledto move downward at an average speed of 58 mms and is pulledupwards at speed of 71 mms 42

35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the roboticarm (a) Position of the microparticles along x-axis The microparticlesmove at average speed of 165 mms along x-axis (b) Motion of themicroparticles along y-axis towards the reference position at an averagespeed of 025 mms (c d e and f) Configuration of the robotic armduring the motion control of the microparticles is represented using qjfor j = 1 7 43

64

List of Figures

36 Kinematic control of a paramagnetic microparticle is achieved usinga permanent magnet attached to the end-effector of the robotic arm(a) A cluster of paramagnetic microparticles is controlled in three-dimensional space towards a reference position (vertical red line) Thecluster is pulled towards the reference position at an average speed of143 mms and 028 mms along x- and y-axis respectively (b) Po-sition of the end-effector of the robotic arm is calculated during theclosed-loop control of the microparticles (c d e and f) Configurationof the robotic arm during the control of the microparticle 44

37 Straight line and sinusoidal trajectory tracking by the controlled mi-croparticles via the controlled magnetic field gradient Waypoints (smallred circles) are defined along the reference trajectory (red lines) andcontrol law (232) is implemented based on (231) 45

38 The configuration of the robotic arm during trajectory tracking of asinusoidal reference trajectory is represented using qj for j = 1 7 46

39 Representative configurations for the robotic are during the tracking ofa sinusoidal trajectory 46

1 Sending message through User Datagram Protocol(UDP) Consists of amessage of 192 bytes that holds the information about the robot stateFirst 24 byts show the cartesian stiffness then 28 bytes that show theaxis stiffness followd by 24 bytes that contains the cartesian dampingthen another 28 bytes with the axis damping information Further 28bytes with the Joints positions values followed by 24 bytes with thecartesian positions Then 4 bytes for the position flag and the messageends with 32 bytes that shows the digital outputs state 70

2 Receiving a message through User Datagram Protocol(UDP) consist-ing of 192 bytes that hold the information about the robot state first28 bytes show the information of the joings angles the next 24 bytesthat show the cartesian position followd by 28 bytes that contains thetorques of the joints then another 28 bytes with the external torquesfurther 12 bytes with the external forces followed by 8 bytes with thetimestamps Then 32 bytes for the digital inputs and the message endswith 32 bytes that shows the digital outputs state 71

3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras 74

65

List of Tables

21 This table shows the definitions of the intrinsic parameters that areextracted 21

22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ithframe of reference 31

31 Setup specifications used in the experiments 40

67

Appendix

1 Communication protocol User Datagram Protocol (UDP)

In this thesis we use UDP as our mean of communication between C and iiwa server(uses java as programming language) the reason behind choosing UDP is because wewanted to transfer the data biderectionally as fast as possible With UDP we cantransfer data in the optimal scenario every 4 millisecond wheras in TCP data thefastest we can get out of the protocol is 8msAn example of how to create a UDPcommunication is shown in the appendix

1001 Sending bytes message from robot server to C

Every 4 ms the windows application which acts as a server in this case sends 192 bytesmessage to the robot server which takes the role of the client in this case The messageis divided into sections as shown in the Figure 1 An example is shown in the appendixto explain how to send byte messages from the robot server or from the c application

The first 24 bytes contains details about the Cartesian stiffness the first 4 bytes aredecoded afterward to extract the stiffness value in X the 4 follow bytes decoded tostiffness value in Y then the next 4 decoded to stiffness value in Z then stiffness valuein A stiffness value in B and finally the stiffness value in C The next 28 bytes holdthe details of the 7 axis stiffness each 4 bytes are decoded to extract the stiffness ofA1 A2 A3 A4 A5 A6 A7 respectively Followed with another 24 bytes which holddetails about Cartesian damping each 4 bytes are decoded to extract the float values

69

List of Tables

Figure 1 Sending message through User Datagram Protocol(UDP) Consists of a mes-sage of 192 bytes that holds the information about the robot state First 24byts show the cartesian stiffness then 28 bytes that show the axis stiffnessfollowd by 24 bytes that contains the cartesian damping then another 28bytes with the axis damping information Further 28 bytes with the Jointspositions values followed by 24 bytes with the cartesian positions Then 4bytes for the position flag and the message ends with 32 bytes that showsthe digital outputs state

of damping in X Y Z A B C respectively Followed with 28 bytes which hold detailsabout Axis damping each 4 bytes are decoded to extract the float values of dampingin A1 A2 A3 A4 A5 A6 A7 respectively Followed with another 28 bytes whichhold details about joint positions each 4 bytes are decoded to extract the float valuesof joint angle (in radian) in A1 A2 A3 A4 A5 A6 A7 respectively Followed with24 bytes which hold details about Cartesian positions each 4 bytes are decoded toextract the float values of XY and Z (in mm) followed by A B and C (in radian)respectively Followed with 4 bytes that is decoded to a float number between 1 and4 defined as follows

1 No movement2 Gravity compensation movement3 Axis movement4 Cartesian movement

Finally the last 32 bytes are used to transfer the values of digital outputs each 2 bytesare decoded to give an integer value either 0 or 1 for each output of the 16 outputs

0 Output of turned off1 Output of turned on

70

1 Communication protocol User Datagram Protocol (UDP)

1002 Receiving byte message from C application in the robot serverOn a different thread every 4 ms the Robot server which acts as a server in this casesends 192 bytes message to the windows application which takes the role of the clientin this case The message is divided into sections as shown in the Figure 2 An exam-ple of how to receive message in c and in the robot message is shown in the appendix

Figure 2 Receiving a message through User Datagram Protocol(UDP) consisting of192 bytes that hold the information about the robot state first 28 bytesshow the information of the joings angles the next 24 bytes that show thecartesian position followd by 28 bytes that contains the torques of the jointsthen another 28 bytes with the external torques further 12 bytes with theexternal forces followed by 8 bytes with the timestamps Then 32 bytes forthe digital inputs and the message ends with 32 bytes that shows the digitaloutputs state

The First 28 bytes shows the Joint Positions These bytes are decoded as explained inthe sending paragraph first 4 bytes have Joints position (in Radian) for A1 A2 A3A4 A5 A6 A7 respectively Followed by 24 bytes that shows the measured Cartesianposition X Y Z (in mm) then A B C (in Radian) respectively Followed by 28 bytesthat contains the measured axis torques (in Nm) for A1 A2 A3 A4 A5 A6 finallyA7 respectively Followed by another 28 bytes that shows the external axis torques(in Nm) for A1 A2 A3 A4 A5 A6 A7 respectively Then 12 bytes that showsthe external forces (in Newton) for Cartesian X Y and Z respectively Followed by 4bytes that is decoded to float that shows the measured torque timestamp (in seconds)Followed by another 4 bytes that is decoded also to another float that shows externaltorque timestamp (in seconds) The last two packets of data contain the details ofthe digital inputsoutputs the first 32 bytes are for 16 digital inputs each 2 bytes aredecoded into an integer of float that is either 0 or 10 Input of turn off1 Input of turn on

71

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

List of Figures

211 Digital image processing steps image acquisition is the first step orprocess of the fundamental steps of digital image processing Imageacquisition could be as simple as being given an image that is alreadyin digital form image enhancement is among the simplest and mostappealing areas of digital image processing The idea behind enhance-ment techniques is to bring out detail that is obscured image restortionis an area that also deals with improving the appearance of an imagemorphological processing deals with tools for extracting image compo-nents that are useful in the representation and description of shapesegmentation procedures partition an image into its constituent partsor objects object recognition is the process that assigns a label suchas vehicle to an object based on its descriptors Finally representa-tion and description almost always follow the output of a segmentationstage which usually is raw pixel data constituting either the boundaryof a region or all the points in the region itself 28

212 C program created to detect the objects (microparticles) the whiteboundry box is drawn around the magnetic particles after detecting theposition of it using the image processing techniques 29

213 Frames assignment of the KUKA LBR iiwa 7 R800 robotic arm Apermanent magnet is attached to the end-effector and its magnetizationvector and z7 are collinear 30

214 Position of the microparticles is mapped onto the configuration of therobotic arm using three homogenous transformations The pose of theend-effector is determined with respect to a frame of reference using0T7 Position of the permanent magnet with respect to the end-effectoris defined using 7TM Position of the microparticle with respect to themagnet is defined using MTp 34

215 Closed-loop motion control of the position of a paramagnetic micropar-ticle using the robotic arm (KUKA iiwa r800) holding the permanentmagnet The particles are contained in water inside a glass tube andtheir position (P) is determined using a camera system and a fea-ture tracking software The control system positions the microparticles(based on the given reference position (Pref) by controlling the positionand orientation of the permenant magnet through the robotic arm 35

31 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluidThe system consist of a robotic arm and a cylindrical NdFeB magnetattached to its end-effector Paramagnetic microparticles with averagediameter of 100 microm are contained inside a glass tube 38

63

List of Figures

32 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluid(a) The system consist of a robotic arm with 7 degrees-of-freedom anda cylindrical NdFeB magnet attached to its end-effector Paramag-netic microparticles (PLAParticles-M-redF-plain from Micromod Par-tikeltechnologie GmbH Rostock-Warnemuende Germany) with aver-age diameter of 100 microm are contained inside a glass tube and posi-tioned in three-dimensional (3D) space using the field gradients (b)The configuration of the robotic arm (indicated using the joint spacecoordinates q) solely control the position of the microparticles in 3Dspace (c) The microparticle follows a path (blue trajectory) control-lably based on the configuration of the robotic arm P1 P2 and P3 arethe position vectors of the microparticles corresponding to the roboticconfigurations q1 q2 and q3 respectively 39

33 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the KUKA LBR iiwa 7R800 robotic arm The black arrow indicates the position of the clusterin the xy-plane whereas the blue line represents the reference posi-tion along z-axis (a) A cluster of microparticle is controlled to movedownward towards a reference position under the influence of controlledmagnetic field gradient and its own weight The magnetic field gradientis controlled via the configuration of the robotic arm (b) The clusteris pulled up towards the reference position using the field gradient 41

34 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the end-effector of theKUKA LBR iiwa 7 R800 robotic arm The microparticles is controlledto move downward at an average speed of 58 mms and is pulledupwards at speed of 71 mms 42

35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the roboticarm (a) Position of the microparticles along x-axis The microparticlesmove at average speed of 165 mms along x-axis (b) Motion of themicroparticles along y-axis towards the reference position at an averagespeed of 025 mms (c d e and f) Configuration of the robotic armduring the motion control of the microparticles is represented using qjfor j = 1 7 43

64

List of Figures

36 Kinematic control of a paramagnetic microparticle is achieved usinga permanent magnet attached to the end-effector of the robotic arm(a) A cluster of paramagnetic microparticles is controlled in three-dimensional space towards a reference position (vertical red line) Thecluster is pulled towards the reference position at an average speed of143 mms and 028 mms along x- and y-axis respectively (b) Po-sition of the end-effector of the robotic arm is calculated during theclosed-loop control of the microparticles (c d e and f) Configurationof the robotic arm during the control of the microparticle 44

37 Straight line and sinusoidal trajectory tracking by the controlled mi-croparticles via the controlled magnetic field gradient Waypoints (smallred circles) are defined along the reference trajectory (red lines) andcontrol law (232) is implemented based on (231) 45

38 The configuration of the robotic arm during trajectory tracking of asinusoidal reference trajectory is represented using qj for j = 1 7 46

39 Representative configurations for the robotic are during the tracking ofa sinusoidal trajectory 46

1 Sending message through User Datagram Protocol(UDP) Consists of amessage of 192 bytes that holds the information about the robot stateFirst 24 byts show the cartesian stiffness then 28 bytes that show theaxis stiffness followd by 24 bytes that contains the cartesian dampingthen another 28 bytes with the axis damping information Further 28bytes with the Joints positions values followed by 24 bytes with thecartesian positions Then 4 bytes for the position flag and the messageends with 32 bytes that shows the digital outputs state 70

2 Receiving a message through User Datagram Protocol(UDP) consist-ing of 192 bytes that hold the information about the robot state first28 bytes show the information of the joings angles the next 24 bytesthat show the cartesian position followd by 28 bytes that contains thetorques of the joints then another 28 bytes with the external torquesfurther 12 bytes with the external forces followed by 8 bytes with thetimestamps Then 32 bytes for the digital inputs and the message endswith 32 bytes that shows the digital outputs state 71

3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras 74

65

List of Tables

21 This table shows the definitions of the intrinsic parameters that areextracted 21

22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ithframe of reference 31

31 Setup specifications used in the experiments 40

67

Appendix

1 Communication protocol User Datagram Protocol (UDP)

In this thesis we use UDP as our mean of communication between C and iiwa server(uses java as programming language) the reason behind choosing UDP is because wewanted to transfer the data biderectionally as fast as possible With UDP we cantransfer data in the optimal scenario every 4 millisecond wheras in TCP data thefastest we can get out of the protocol is 8msAn example of how to create a UDPcommunication is shown in the appendix

1001 Sending bytes message from robot server to C

Every 4 ms the windows application which acts as a server in this case sends 192 bytesmessage to the robot server which takes the role of the client in this case The messageis divided into sections as shown in the Figure 1 An example is shown in the appendixto explain how to send byte messages from the robot server or from the c application

The first 24 bytes contains details about the Cartesian stiffness the first 4 bytes aredecoded afterward to extract the stiffness value in X the 4 follow bytes decoded tostiffness value in Y then the next 4 decoded to stiffness value in Z then stiffness valuein A stiffness value in B and finally the stiffness value in C The next 28 bytes holdthe details of the 7 axis stiffness each 4 bytes are decoded to extract the stiffness ofA1 A2 A3 A4 A5 A6 A7 respectively Followed with another 24 bytes which holddetails about Cartesian damping each 4 bytes are decoded to extract the float values

69

List of Tables

Figure 1 Sending message through User Datagram Protocol(UDP) Consists of a mes-sage of 192 bytes that holds the information about the robot state First 24byts show the cartesian stiffness then 28 bytes that show the axis stiffnessfollowd by 24 bytes that contains the cartesian damping then another 28bytes with the axis damping information Further 28 bytes with the Jointspositions values followed by 24 bytes with the cartesian positions Then 4bytes for the position flag and the message ends with 32 bytes that showsthe digital outputs state

of damping in X Y Z A B C respectively Followed with 28 bytes which hold detailsabout Axis damping each 4 bytes are decoded to extract the float values of dampingin A1 A2 A3 A4 A5 A6 A7 respectively Followed with another 28 bytes whichhold details about joint positions each 4 bytes are decoded to extract the float valuesof joint angle (in radian) in A1 A2 A3 A4 A5 A6 A7 respectively Followed with24 bytes which hold details about Cartesian positions each 4 bytes are decoded toextract the float values of XY and Z (in mm) followed by A B and C (in radian)respectively Followed with 4 bytes that is decoded to a float number between 1 and4 defined as follows

1 No movement2 Gravity compensation movement3 Axis movement4 Cartesian movement

Finally the last 32 bytes are used to transfer the values of digital outputs each 2 bytesare decoded to give an integer value either 0 or 1 for each output of the 16 outputs

0 Output of turned off1 Output of turned on

70

1 Communication protocol User Datagram Protocol (UDP)

1002 Receiving byte message from C application in the robot serverOn a different thread every 4 ms the Robot server which acts as a server in this casesends 192 bytes message to the windows application which takes the role of the clientin this case The message is divided into sections as shown in the Figure 2 An exam-ple of how to receive message in c and in the robot message is shown in the appendix

Figure 2 Receiving a message through User Datagram Protocol(UDP) consisting of192 bytes that hold the information about the robot state first 28 bytesshow the information of the joings angles the next 24 bytes that show thecartesian position followd by 28 bytes that contains the torques of the jointsthen another 28 bytes with the external torques further 12 bytes with theexternal forces followed by 8 bytes with the timestamps Then 32 bytes forthe digital inputs and the message ends with 32 bytes that shows the digitaloutputs state

The First 28 bytes shows the Joint Positions These bytes are decoded as explained inthe sending paragraph first 4 bytes have Joints position (in Radian) for A1 A2 A3A4 A5 A6 A7 respectively Followed by 24 bytes that shows the measured Cartesianposition X Y Z (in mm) then A B C (in Radian) respectively Followed by 28 bytesthat contains the measured axis torques (in Nm) for A1 A2 A3 A4 A5 A6 finallyA7 respectively Followed by another 28 bytes that shows the external axis torques(in Nm) for A1 A2 A3 A4 A5 A6 A7 respectively Then 12 bytes that showsthe external forces (in Newton) for Cartesian X Y and Z respectively Followed by 4bytes that is decoded to float that shows the measured torque timestamp (in seconds)Followed by another 4 bytes that is decoded also to another float that shows externaltorque timestamp (in seconds) The last two packets of data contain the details ofthe digital inputsoutputs the first 32 bytes are for 16 digital inputs each 2 bytes aredecoded into an integer of float that is either 0 or 10 Input of turn off1 Input of turn on

71

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

List of Figures

32 A permanent magnet-based actuation system is used to control andposition paramagnetic microparticles in a low Reynolds number fluid(a) The system consist of a robotic arm with 7 degrees-of-freedom anda cylindrical NdFeB magnet attached to its end-effector Paramag-netic microparticles (PLAParticles-M-redF-plain from Micromod Par-tikeltechnologie GmbH Rostock-Warnemuende Germany) with aver-age diameter of 100 microm are contained inside a glass tube and posi-tioned in three-dimensional (3D) space using the field gradients (b)The configuration of the robotic arm (indicated using the joint spacecoordinates q) solely control the position of the microparticles in 3Dspace (c) The microparticle follows a path (blue trajectory) control-lably based on the configuration of the robotic arm P1 P2 and P3 arethe position vectors of the microparticles corresponding to the roboticconfigurations q1 q2 and q3 respectively 39

33 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the KUKA LBR iiwa 7R800 robotic arm The black arrow indicates the position of the clusterin the xy-plane whereas the blue line represents the reference posi-tion along z-axis (a) A cluster of microparticle is controlled to movedownward towards a reference position under the influence of controlledmagnetic field gradient and its own weight The magnetic field gradientis controlled via the configuration of the robotic arm (b) The clusteris pulled up towards the reference position using the field gradient 41

34 Representative motion control of a cluster of paramagnetic micropar-ticles using a permanent magnet attached to the end-effector of theKUKA LBR iiwa 7 R800 robotic arm The microparticles is controlledto move downward at an average speed of 58 mms and is pulledupwards at speed of 71 mms 42

35 Kinematic control of a cluster of paramagnetic microparticles is achievedusing a permanent magnet attached to the end-effector of the roboticarm (a) Position of the microparticles along x-axis The microparticlesmove at average speed of 165 mms along x-axis (b) Motion of themicroparticles along y-axis towards the reference position at an averagespeed of 025 mms (c d e and f) Configuration of the robotic armduring the motion control of the microparticles is represented using qjfor j = 1 7 43

64

List of Figures

36 Kinematic control of a paramagnetic microparticle is achieved usinga permanent magnet attached to the end-effector of the robotic arm(a) A cluster of paramagnetic microparticles is controlled in three-dimensional space towards a reference position (vertical red line) Thecluster is pulled towards the reference position at an average speed of143 mms and 028 mms along x- and y-axis respectively (b) Po-sition of the end-effector of the robotic arm is calculated during theclosed-loop control of the microparticles (c d e and f) Configurationof the robotic arm during the control of the microparticle 44

37 Straight line and sinusoidal trajectory tracking by the controlled mi-croparticles via the controlled magnetic field gradient Waypoints (smallred circles) are defined along the reference trajectory (red lines) andcontrol law (232) is implemented based on (231) 45

38 The configuration of the robotic arm during trajectory tracking of asinusoidal reference trajectory is represented using qj for j = 1 7 46

39 Representative configurations for the robotic are during the tracking ofa sinusoidal trajectory 46

1 Sending message through User Datagram Protocol(UDP) Consists of amessage of 192 bytes that holds the information about the robot stateFirst 24 byts show the cartesian stiffness then 28 bytes that show theaxis stiffness followd by 24 bytes that contains the cartesian dampingthen another 28 bytes with the axis damping information Further 28bytes with the Joints positions values followed by 24 bytes with thecartesian positions Then 4 bytes for the position flag and the messageends with 32 bytes that shows the digital outputs state 70

2 Receiving a message through User Datagram Protocol(UDP) consist-ing of 192 bytes that hold the information about the robot state first28 bytes show the information of the joings angles the next 24 bytesthat show the cartesian position followd by 28 bytes that contains thetorques of the joints then another 28 bytes with the external torquesfurther 12 bytes with the external forces followed by 8 bytes with thetimestamps Then 32 bytes for the digital inputs and the message endswith 32 bytes that shows the digital outputs state 71

3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras 74

65

List of Tables

21 This table shows the definitions of the intrinsic parameters that areextracted 21

22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ithframe of reference 31

31 Setup specifications used in the experiments 40

67

Appendix

1 Communication protocol User Datagram Protocol (UDP)

In this thesis we use UDP as our mean of communication between C and iiwa server(uses java as programming language) the reason behind choosing UDP is because wewanted to transfer the data biderectionally as fast as possible With UDP we cantransfer data in the optimal scenario every 4 millisecond wheras in TCP data thefastest we can get out of the protocol is 8msAn example of how to create a UDPcommunication is shown in the appendix

1001 Sending bytes message from robot server to C

Every 4 ms the windows application which acts as a server in this case sends 192 bytesmessage to the robot server which takes the role of the client in this case The messageis divided into sections as shown in the Figure 1 An example is shown in the appendixto explain how to send byte messages from the robot server or from the c application

The first 24 bytes contains details about the Cartesian stiffness the first 4 bytes aredecoded afterward to extract the stiffness value in X the 4 follow bytes decoded tostiffness value in Y then the next 4 decoded to stiffness value in Z then stiffness valuein A stiffness value in B and finally the stiffness value in C The next 28 bytes holdthe details of the 7 axis stiffness each 4 bytes are decoded to extract the stiffness ofA1 A2 A3 A4 A5 A6 A7 respectively Followed with another 24 bytes which holddetails about Cartesian damping each 4 bytes are decoded to extract the float values

69

List of Tables

Figure 1 Sending message through User Datagram Protocol(UDP) Consists of a mes-sage of 192 bytes that holds the information about the robot state First 24byts show the cartesian stiffness then 28 bytes that show the axis stiffnessfollowd by 24 bytes that contains the cartesian damping then another 28bytes with the axis damping information Further 28 bytes with the Jointspositions values followed by 24 bytes with the cartesian positions Then 4bytes for the position flag and the message ends with 32 bytes that showsthe digital outputs state

of damping in X Y Z A B C respectively Followed with 28 bytes which hold detailsabout Axis damping each 4 bytes are decoded to extract the float values of dampingin A1 A2 A3 A4 A5 A6 A7 respectively Followed with another 28 bytes whichhold details about joint positions each 4 bytes are decoded to extract the float valuesof joint angle (in radian) in A1 A2 A3 A4 A5 A6 A7 respectively Followed with24 bytes which hold details about Cartesian positions each 4 bytes are decoded toextract the float values of XY and Z (in mm) followed by A B and C (in radian)respectively Followed with 4 bytes that is decoded to a float number between 1 and4 defined as follows

1 No movement2 Gravity compensation movement3 Axis movement4 Cartesian movement

Finally the last 32 bytes are used to transfer the values of digital outputs each 2 bytesare decoded to give an integer value either 0 or 1 for each output of the 16 outputs

0 Output of turned off1 Output of turned on

70

1 Communication protocol User Datagram Protocol (UDP)

1002 Receiving byte message from C application in the robot serverOn a different thread every 4 ms the Robot server which acts as a server in this casesends 192 bytes message to the windows application which takes the role of the clientin this case The message is divided into sections as shown in the Figure 2 An exam-ple of how to receive message in c and in the robot message is shown in the appendix

Figure 2 Receiving a message through User Datagram Protocol(UDP) consisting of192 bytes that hold the information about the robot state first 28 bytesshow the information of the joings angles the next 24 bytes that show thecartesian position followd by 28 bytes that contains the torques of the jointsthen another 28 bytes with the external torques further 12 bytes with theexternal forces followed by 8 bytes with the timestamps Then 32 bytes forthe digital inputs and the message ends with 32 bytes that shows the digitaloutputs state

The First 28 bytes shows the Joint Positions These bytes are decoded as explained inthe sending paragraph first 4 bytes have Joints position (in Radian) for A1 A2 A3A4 A5 A6 A7 respectively Followed by 24 bytes that shows the measured Cartesianposition X Y Z (in mm) then A B C (in Radian) respectively Followed by 28 bytesthat contains the measured axis torques (in Nm) for A1 A2 A3 A4 A5 A6 finallyA7 respectively Followed by another 28 bytes that shows the external axis torques(in Nm) for A1 A2 A3 A4 A5 A6 A7 respectively Then 12 bytes that showsthe external forces (in Newton) for Cartesian X Y and Z respectively Followed by 4bytes that is decoded to float that shows the measured torque timestamp (in seconds)Followed by another 4 bytes that is decoded also to another float that shows externaltorque timestamp (in seconds) The last two packets of data contain the details ofthe digital inputsoutputs the first 32 bytes are for 16 digital inputs each 2 bytes aredecoded into an integer of float that is either 0 or 10 Input of turn off1 Input of turn on

71

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

List of Figures

36 Kinematic control of a paramagnetic microparticle is achieved usinga permanent magnet attached to the end-effector of the robotic arm(a) A cluster of paramagnetic microparticles is controlled in three-dimensional space towards a reference position (vertical red line) Thecluster is pulled towards the reference position at an average speed of143 mms and 028 mms along x- and y-axis respectively (b) Po-sition of the end-effector of the robotic arm is calculated during theclosed-loop control of the microparticles (c d e and f) Configurationof the robotic arm during the control of the microparticle 44

37 Straight line and sinusoidal trajectory tracking by the controlled mi-croparticles via the controlled magnetic field gradient Waypoints (smallred circles) are defined along the reference trajectory (red lines) andcontrol law (232) is implemented based on (231) 45

38 The configuration of the robotic arm during trajectory tracking of asinusoidal reference trajectory is represented using qj for j = 1 7 46

39 Representative configurations for the robotic are during the tracking ofa sinusoidal trajectory 46

1 Sending message through User Datagram Protocol(UDP) Consists of amessage of 192 bytes that holds the information about the robot stateFirst 24 byts show the cartesian stiffness then 28 bytes that show theaxis stiffness followd by 24 bytes that contains the cartesian dampingthen another 28 bytes with the axis damping information Further 28bytes with the Joints positions values followed by 24 bytes with thecartesian positions Then 4 bytes for the position flag and the messageends with 32 bytes that shows the digital outputs state 70

2 Receiving a message through User Datagram Protocol(UDP) consist-ing of 192 bytes that hold the information about the robot state first28 bytes show the information of the joings angles the next 24 bytesthat show the cartesian position followd by 28 bytes that contains thetorques of the joints then another 28 bytes with the external torquesfurther 12 bytes with the external forces followed by 8 bytes with thetimestamps Then 32 bytes for the digital inputs and the message endswith 32 bytes that shows the digital outputs state 71

3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras 74

65

List of Tables

21 This table shows the definitions of the intrinsic parameters that areextracted 21

22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ithframe of reference 31

31 Setup specifications used in the experiments 40

67

Appendix

1 Communication protocol User Datagram Protocol (UDP)

In this thesis we use UDP as our mean of communication between C and iiwa server(uses java as programming language) the reason behind choosing UDP is because wewanted to transfer the data biderectionally as fast as possible With UDP we cantransfer data in the optimal scenario every 4 millisecond wheras in TCP data thefastest we can get out of the protocol is 8msAn example of how to create a UDPcommunication is shown in the appendix

1001 Sending bytes message from robot server to C

Every 4 ms the windows application which acts as a server in this case sends 192 bytesmessage to the robot server which takes the role of the client in this case The messageis divided into sections as shown in the Figure 1 An example is shown in the appendixto explain how to send byte messages from the robot server or from the c application

The first 24 bytes contains details about the Cartesian stiffness the first 4 bytes aredecoded afterward to extract the stiffness value in X the 4 follow bytes decoded tostiffness value in Y then the next 4 decoded to stiffness value in Z then stiffness valuein A stiffness value in B and finally the stiffness value in C The next 28 bytes holdthe details of the 7 axis stiffness each 4 bytes are decoded to extract the stiffness ofA1 A2 A3 A4 A5 A6 A7 respectively Followed with another 24 bytes which holddetails about Cartesian damping each 4 bytes are decoded to extract the float values

69

List of Tables

Figure 1 Sending message through User Datagram Protocol(UDP) Consists of a mes-sage of 192 bytes that holds the information about the robot state First 24byts show the cartesian stiffness then 28 bytes that show the axis stiffnessfollowd by 24 bytes that contains the cartesian damping then another 28bytes with the axis damping information Further 28 bytes with the Jointspositions values followed by 24 bytes with the cartesian positions Then 4bytes for the position flag and the message ends with 32 bytes that showsthe digital outputs state

of damping in X Y Z A B C respectively Followed with 28 bytes which hold detailsabout Axis damping each 4 bytes are decoded to extract the float values of dampingin A1 A2 A3 A4 A5 A6 A7 respectively Followed with another 28 bytes whichhold details about joint positions each 4 bytes are decoded to extract the float valuesof joint angle (in radian) in A1 A2 A3 A4 A5 A6 A7 respectively Followed with24 bytes which hold details about Cartesian positions each 4 bytes are decoded toextract the float values of XY and Z (in mm) followed by A B and C (in radian)respectively Followed with 4 bytes that is decoded to a float number between 1 and4 defined as follows

1 No movement2 Gravity compensation movement3 Axis movement4 Cartesian movement

Finally the last 32 bytes are used to transfer the values of digital outputs each 2 bytesare decoded to give an integer value either 0 or 1 for each output of the 16 outputs

0 Output of turned off1 Output of turned on

70

1 Communication protocol User Datagram Protocol (UDP)

1002 Receiving byte message from C application in the robot serverOn a different thread every 4 ms the Robot server which acts as a server in this casesends 192 bytes message to the windows application which takes the role of the clientin this case The message is divided into sections as shown in the Figure 2 An exam-ple of how to receive message in c and in the robot message is shown in the appendix

Figure 2 Receiving a message through User Datagram Protocol(UDP) consisting of192 bytes that hold the information about the robot state first 28 bytesshow the information of the joings angles the next 24 bytes that show thecartesian position followd by 28 bytes that contains the torques of the jointsthen another 28 bytes with the external torques further 12 bytes with theexternal forces followed by 8 bytes with the timestamps Then 32 bytes forthe digital inputs and the message ends with 32 bytes that shows the digitaloutputs state

The First 28 bytes shows the Joint Positions These bytes are decoded as explained inthe sending paragraph first 4 bytes have Joints position (in Radian) for A1 A2 A3A4 A5 A6 A7 respectively Followed by 24 bytes that shows the measured Cartesianposition X Y Z (in mm) then A B C (in Radian) respectively Followed by 28 bytesthat contains the measured axis torques (in Nm) for A1 A2 A3 A4 A5 A6 finallyA7 respectively Followed by another 28 bytes that shows the external axis torques(in Nm) for A1 A2 A3 A4 A5 A6 A7 respectively Then 12 bytes that showsthe external forces (in Newton) for Cartesian X Y and Z respectively Followed by 4bytes that is decoded to float that shows the measured torque timestamp (in seconds)Followed by another 4 bytes that is decoded also to another float that shows externaltorque timestamp (in seconds) The last two packets of data contain the details ofthe digital inputsoutputs the first 32 bytes are for 16 digital inputs each 2 bytes aredecoded into an integer of float that is either 0 or 10 Input of turn off1 Input of turn on

71

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

List of Tables

21 This table shows the definitions of the intrinsic parameters that areextracted 21

22 Denavit-Hartenberg (DH) parameters of the KUKA LBR iiwa 7 R800robotic arm αiminus1 aiminus1 di and θi are the DH parameters of the ithframe of reference 31

31 Setup specifications used in the experiments 40

67

Appendix

1 Communication protocol User Datagram Protocol (UDP)

In this thesis we use UDP as our mean of communication between C and iiwa server(uses java as programming language) the reason behind choosing UDP is because wewanted to transfer the data biderectionally as fast as possible With UDP we cantransfer data in the optimal scenario every 4 millisecond wheras in TCP data thefastest we can get out of the protocol is 8msAn example of how to create a UDPcommunication is shown in the appendix

1001 Sending bytes message from robot server to C

Every 4 ms the windows application which acts as a server in this case sends 192 bytesmessage to the robot server which takes the role of the client in this case The messageis divided into sections as shown in the Figure 1 An example is shown in the appendixto explain how to send byte messages from the robot server or from the c application

The first 24 bytes contains details about the Cartesian stiffness the first 4 bytes aredecoded afterward to extract the stiffness value in X the 4 follow bytes decoded tostiffness value in Y then the next 4 decoded to stiffness value in Z then stiffness valuein A stiffness value in B and finally the stiffness value in C The next 28 bytes holdthe details of the 7 axis stiffness each 4 bytes are decoded to extract the stiffness ofA1 A2 A3 A4 A5 A6 A7 respectively Followed with another 24 bytes which holddetails about Cartesian damping each 4 bytes are decoded to extract the float values

69

List of Tables

Figure 1 Sending message through User Datagram Protocol(UDP) Consists of a mes-sage of 192 bytes that holds the information about the robot state First 24byts show the cartesian stiffness then 28 bytes that show the axis stiffnessfollowd by 24 bytes that contains the cartesian damping then another 28bytes with the axis damping information Further 28 bytes with the Jointspositions values followed by 24 bytes with the cartesian positions Then 4bytes for the position flag and the message ends with 32 bytes that showsthe digital outputs state

of damping in X Y Z A B C respectively Followed with 28 bytes which hold detailsabout Axis damping each 4 bytes are decoded to extract the float values of dampingin A1 A2 A3 A4 A5 A6 A7 respectively Followed with another 28 bytes whichhold details about joint positions each 4 bytes are decoded to extract the float valuesof joint angle (in radian) in A1 A2 A3 A4 A5 A6 A7 respectively Followed with24 bytes which hold details about Cartesian positions each 4 bytes are decoded toextract the float values of XY and Z (in mm) followed by A B and C (in radian)respectively Followed with 4 bytes that is decoded to a float number between 1 and4 defined as follows

1 No movement2 Gravity compensation movement3 Axis movement4 Cartesian movement

Finally the last 32 bytes are used to transfer the values of digital outputs each 2 bytesare decoded to give an integer value either 0 or 1 for each output of the 16 outputs

0 Output of turned off1 Output of turned on

70

1 Communication protocol User Datagram Protocol (UDP)

1002 Receiving byte message from C application in the robot serverOn a different thread every 4 ms the Robot server which acts as a server in this casesends 192 bytes message to the windows application which takes the role of the clientin this case The message is divided into sections as shown in the Figure 2 An exam-ple of how to receive message in c and in the robot message is shown in the appendix

Figure 2 Receiving a message through User Datagram Protocol(UDP) consisting of192 bytes that hold the information about the robot state first 28 bytesshow the information of the joings angles the next 24 bytes that show thecartesian position followd by 28 bytes that contains the torques of the jointsthen another 28 bytes with the external torques further 12 bytes with theexternal forces followed by 8 bytes with the timestamps Then 32 bytes forthe digital inputs and the message ends with 32 bytes that shows the digitaloutputs state

The First 28 bytes shows the Joint Positions These bytes are decoded as explained inthe sending paragraph first 4 bytes have Joints position (in Radian) for A1 A2 A3A4 A5 A6 A7 respectively Followed by 24 bytes that shows the measured Cartesianposition X Y Z (in mm) then A B C (in Radian) respectively Followed by 28 bytesthat contains the measured axis torques (in Nm) for A1 A2 A3 A4 A5 A6 finallyA7 respectively Followed by another 28 bytes that shows the external axis torques(in Nm) for A1 A2 A3 A4 A5 A6 A7 respectively Then 12 bytes that showsthe external forces (in Newton) for Cartesian X Y and Z respectively Followed by 4bytes that is decoded to float that shows the measured torque timestamp (in seconds)Followed by another 4 bytes that is decoded also to another float that shows externaltorque timestamp (in seconds) The last two packets of data contain the details ofthe digital inputsoutputs the first 32 bytes are for 16 digital inputs each 2 bytes aredecoded into an integer of float that is either 0 or 10 Input of turn off1 Input of turn on

71

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

Appendix

1 Communication protocol User Datagram Protocol (UDP)

In this thesis we use UDP as our mean of communication between C and iiwa server(uses java as programming language) the reason behind choosing UDP is because wewanted to transfer the data biderectionally as fast as possible With UDP we cantransfer data in the optimal scenario every 4 millisecond wheras in TCP data thefastest we can get out of the protocol is 8msAn example of how to create a UDPcommunication is shown in the appendix

1001 Sending bytes message from robot server to C

Every 4 ms the windows application which acts as a server in this case sends 192 bytesmessage to the robot server which takes the role of the client in this case The messageis divided into sections as shown in the Figure 1 An example is shown in the appendixto explain how to send byte messages from the robot server or from the c application

The first 24 bytes contains details about the Cartesian stiffness the first 4 bytes aredecoded afterward to extract the stiffness value in X the 4 follow bytes decoded tostiffness value in Y then the next 4 decoded to stiffness value in Z then stiffness valuein A stiffness value in B and finally the stiffness value in C The next 28 bytes holdthe details of the 7 axis stiffness each 4 bytes are decoded to extract the stiffness ofA1 A2 A3 A4 A5 A6 A7 respectively Followed with another 24 bytes which holddetails about Cartesian damping each 4 bytes are decoded to extract the float values

69

List of Tables

Figure 1 Sending message through User Datagram Protocol(UDP) Consists of a mes-sage of 192 bytes that holds the information about the robot state First 24byts show the cartesian stiffness then 28 bytes that show the axis stiffnessfollowd by 24 bytes that contains the cartesian damping then another 28bytes with the axis damping information Further 28 bytes with the Jointspositions values followed by 24 bytes with the cartesian positions Then 4bytes for the position flag and the message ends with 32 bytes that showsthe digital outputs state

of damping in X Y Z A B C respectively Followed with 28 bytes which hold detailsabout Axis damping each 4 bytes are decoded to extract the float values of dampingin A1 A2 A3 A4 A5 A6 A7 respectively Followed with another 28 bytes whichhold details about joint positions each 4 bytes are decoded to extract the float valuesof joint angle (in radian) in A1 A2 A3 A4 A5 A6 A7 respectively Followed with24 bytes which hold details about Cartesian positions each 4 bytes are decoded toextract the float values of XY and Z (in mm) followed by A B and C (in radian)respectively Followed with 4 bytes that is decoded to a float number between 1 and4 defined as follows

1 No movement2 Gravity compensation movement3 Axis movement4 Cartesian movement

Finally the last 32 bytes are used to transfer the values of digital outputs each 2 bytesare decoded to give an integer value either 0 or 1 for each output of the 16 outputs

0 Output of turned off1 Output of turned on

70

1 Communication protocol User Datagram Protocol (UDP)

1002 Receiving byte message from C application in the robot serverOn a different thread every 4 ms the Robot server which acts as a server in this casesends 192 bytes message to the windows application which takes the role of the clientin this case The message is divided into sections as shown in the Figure 2 An exam-ple of how to receive message in c and in the robot message is shown in the appendix

Figure 2 Receiving a message through User Datagram Protocol(UDP) consisting of192 bytes that hold the information about the robot state first 28 bytesshow the information of the joings angles the next 24 bytes that show thecartesian position followd by 28 bytes that contains the torques of the jointsthen another 28 bytes with the external torques further 12 bytes with theexternal forces followed by 8 bytes with the timestamps Then 32 bytes forthe digital inputs and the message ends with 32 bytes that shows the digitaloutputs state

The First 28 bytes shows the Joint Positions These bytes are decoded as explained inthe sending paragraph first 4 bytes have Joints position (in Radian) for A1 A2 A3A4 A5 A6 A7 respectively Followed by 24 bytes that shows the measured Cartesianposition X Y Z (in mm) then A B C (in Radian) respectively Followed by 28 bytesthat contains the measured axis torques (in Nm) for A1 A2 A3 A4 A5 A6 finallyA7 respectively Followed by another 28 bytes that shows the external axis torques(in Nm) for A1 A2 A3 A4 A5 A6 A7 respectively Then 12 bytes that showsthe external forces (in Newton) for Cartesian X Y and Z respectively Followed by 4bytes that is decoded to float that shows the measured torque timestamp (in seconds)Followed by another 4 bytes that is decoded also to another float that shows externaltorque timestamp (in seconds) The last two packets of data contain the details ofthe digital inputsoutputs the first 32 bytes are for 16 digital inputs each 2 bytes aredecoded into an integer of float that is either 0 or 10 Input of turn off1 Input of turn on

71

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

List of Tables

Figure 1 Sending message through User Datagram Protocol(UDP) Consists of a mes-sage of 192 bytes that holds the information about the robot state First 24byts show the cartesian stiffness then 28 bytes that show the axis stiffnessfollowd by 24 bytes that contains the cartesian damping then another 28bytes with the axis damping information Further 28 bytes with the Jointspositions values followed by 24 bytes with the cartesian positions Then 4bytes for the position flag and the message ends with 32 bytes that showsthe digital outputs state

of damping in X Y Z A B C respectively Followed with 28 bytes which hold detailsabout Axis damping each 4 bytes are decoded to extract the float values of dampingin A1 A2 A3 A4 A5 A6 A7 respectively Followed with another 28 bytes whichhold details about joint positions each 4 bytes are decoded to extract the float valuesof joint angle (in radian) in A1 A2 A3 A4 A5 A6 A7 respectively Followed with24 bytes which hold details about Cartesian positions each 4 bytes are decoded toextract the float values of XY and Z (in mm) followed by A B and C (in radian)respectively Followed with 4 bytes that is decoded to a float number between 1 and4 defined as follows

1 No movement2 Gravity compensation movement3 Axis movement4 Cartesian movement

Finally the last 32 bytes are used to transfer the values of digital outputs each 2 bytesare decoded to give an integer value either 0 or 1 for each output of the 16 outputs

0 Output of turned off1 Output of turned on

70

1 Communication protocol User Datagram Protocol (UDP)

1002 Receiving byte message from C application in the robot serverOn a different thread every 4 ms the Robot server which acts as a server in this casesends 192 bytes message to the windows application which takes the role of the clientin this case The message is divided into sections as shown in the Figure 2 An exam-ple of how to receive message in c and in the robot message is shown in the appendix

Figure 2 Receiving a message through User Datagram Protocol(UDP) consisting of192 bytes that hold the information about the robot state first 28 bytesshow the information of the joings angles the next 24 bytes that show thecartesian position followd by 28 bytes that contains the torques of the jointsthen another 28 bytes with the external torques further 12 bytes with theexternal forces followed by 8 bytes with the timestamps Then 32 bytes forthe digital inputs and the message ends with 32 bytes that shows the digitaloutputs state

The First 28 bytes shows the Joint Positions These bytes are decoded as explained inthe sending paragraph first 4 bytes have Joints position (in Radian) for A1 A2 A3A4 A5 A6 A7 respectively Followed by 24 bytes that shows the measured Cartesianposition X Y Z (in mm) then A B C (in Radian) respectively Followed by 28 bytesthat contains the measured axis torques (in Nm) for A1 A2 A3 A4 A5 A6 finallyA7 respectively Followed by another 28 bytes that shows the external axis torques(in Nm) for A1 A2 A3 A4 A5 A6 A7 respectively Then 12 bytes that showsthe external forces (in Newton) for Cartesian X Y and Z respectively Followed by 4bytes that is decoded to float that shows the measured torque timestamp (in seconds)Followed by another 4 bytes that is decoded also to another float that shows externaltorque timestamp (in seconds) The last two packets of data contain the details ofthe digital inputsoutputs the first 32 bytes are for 16 digital inputs each 2 bytes aredecoded into an integer of float that is either 0 or 10 Input of turn off1 Input of turn on

71

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

1 Communication protocol User Datagram Protocol (UDP)

1002 Receiving byte message from C application in the robot serverOn a different thread every 4 ms the Robot server which acts as a server in this casesends 192 bytes message to the windows application which takes the role of the clientin this case The message is divided into sections as shown in the Figure 2 An exam-ple of how to receive message in c and in the robot message is shown in the appendix

Figure 2 Receiving a message through User Datagram Protocol(UDP) consisting of192 bytes that hold the information about the robot state first 28 bytesshow the information of the joings angles the next 24 bytes that show thecartesian position followd by 28 bytes that contains the torques of the jointsthen another 28 bytes with the external torques further 12 bytes with theexternal forces followed by 8 bytes with the timestamps Then 32 bytes forthe digital inputs and the message ends with 32 bytes that shows the digitaloutputs state

The First 28 bytes shows the Joint Positions These bytes are decoded as explained inthe sending paragraph first 4 bytes have Joints position (in Radian) for A1 A2 A3A4 A5 A6 A7 respectively Followed by 24 bytes that shows the measured Cartesianposition X Y Z (in mm) then A B C (in Radian) respectively Followed by 28 bytesthat contains the measured axis torques (in Nm) for A1 A2 A3 A4 A5 A6 finallyA7 respectively Followed by another 28 bytes that shows the external axis torques(in Nm) for A1 A2 A3 A4 A5 A6 A7 respectively Then 12 bytes that showsthe external forces (in Newton) for Cartesian X Y and Z respectively Followed by 4bytes that is decoded to float that shows the measured torque timestamp (in seconds)Followed by another 4 bytes that is decoded also to another float that shows externaltorque timestamp (in seconds) The last two packets of data contain the details ofthe digital inputsoutputs the first 32 bytes are for 16 digital inputs each 2 bytes aredecoded into an integer of float that is either 0 or 10 Input of turn off1 Input of turn on

71

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

List of Tables

assigned to each an every input starting from input 1 to 16 respectively The nextand last 32 bytes are for the 16 digital output each 2 bytes are also decoded into aninteger float 0 or 10 Output of turn off1 Output of turn onassigned to every output starting from output 1 to 16 respectively

2 Creating a UDP communication in C application

An example to show how to create a UDP communication in C application with anyserver an ip address and port from the server should be included as a static ip in thehost pc and a different port for sending from c application to the server should bedefined too

p r i va t e void CreateUDPSocket ( )

Create a UDP Socket f o r XPCminusCommunicationsend_data_xpc = new byte [ 8 8 ] receive_data_xpc = new byte [ 6 4 ]

ipep_xpc = new IPEndPoint( IPAddress Parse ( 1 9 2 1 6 8 2 0 7 ) 9050 ) ipep_xpc = new IPEndPoint( IPAddress Any 30200) udp_xpc_socket =new UdpClient ( ipep_xpc ) udp_xpc_socket C l i en t Rece i v eBu f f e rS i z e = 64 udp_xpc_socket C l i en t ReceiveTimeout = 10 sender_xpc2 = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 30010) RobotSendingPort = newIPEndPoint ( IPAddress Parse ( 1 7 2 3 1 1 1 4 7 ) 18000)

72

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

3 Recieving a byte message using udp in java and C

3 Recieving a byte message using udp in java and C

To receive through java or c this command should be used [42] [41]in JavamySocket r e c e i v e ( )

in Cudp_xpc_socket Rece ive ( r e f RobotSendingPort )

4 Sending a byte message using udp in java and C

To Send through java or c this command should be used [42] [41]in JavamySocket send ( bytemessage length port )

in Cudp_xpc_socket Send ( bytemessage length port )

5 Image processing application example

A feature tracking algorithm using emguCV library and C as the programminglanguage is given belowpr i va t e ImageltGray bytegt ImageProcessingXY

( ImageltGray bytegt Cameraimg )

camera img to grayImageltGray bytegt grayImage =Cameraimg ConvertltGray byte gt()

Convert img to binarygrayImage ROI =new Rectangle (0 0 600 300 ) ImageltGray bytegt binaryImg = grayImage

73

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

List of Tables

Figure 3 a c application was created for this thesis the graphical user interfaceconsists of two live views images for two cameras

ThresholdBinaryInv (new Gray (90 ) new Gray ( 2 5 5 ) ) binaryImg D i l a t e ( 1 5 ) binaryImg Save ( binaryImg png )

Find ContoursContourltPointgt contours =

binaryImg FindContours ( )

Find the Bigges t contourdouble BiggestContourArea = 0 ContourltPointgt Largestcontour = contours

i f ( contours = n u l l )

74

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

5 Image processing application example

f i n d the Clus te r Rectangle

double smal lestX = contours BoundingRectangle Le f t double smal lestY = contours BoundingRectangle Top double l a rge s tX = contours BoundingRectangle Right double l a rge s tY = contours BoundingRectangle Bottom

whi l e ( contours = n u l l )

Cameraimg Draw ( )double currentArea = contours Area i f ( currentArea gt BiggestContourArea )

Largestcontour = contours BiggestContourArea = currentArea

i f ( contours BoundingRectangle Le f t lt smal lestX )

smal lestX = contours BoundingRectangle Le f t i f ( contours BoundingRectangle Right gt large s tX )

la rge s tX = contours BoundingRectangle Right i f ( contours BoundingRectangle Top lt smal lestY )

smal lestY = contours BoundingRectangle Top i f ( contours BoundingRectangle Bottom gt large s tY )

la rge s tY = contours BoundingRectangle Bottom contours = contours HNext

Plot b i g g e s t contourCameraimg Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) ) new Gray (255) 3 )

XYshowImage = Cameraimg ConvertltRgb byte gt() XYshowImage Draw(new Rectangle ( ( i n t ) ( smal lestX ) ( i n t ) smallestY

( i n t ) ( l a rge s tX minus smal lestX ) ( i n t ) ( l a rge s tY minus smal lestY ) )

75

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example

List of Tables

new Rgb(255 0 0 ) 3 )

dummyimg Draw( s r e f font new Point ( Largestcontour BoundingRectangle Locat ion X + 20 Largestcontour BoundingRectangle Locat ion Y + 30) new Rgb(255 0 0 ) ) currentPosX = ( i n t ) ( ( smal lestX +( ( la rge s tX minus smal lestX ) 2 ) ) ) currentPosY = ( i n t ) ( smal lestY +( ( la rge s tY minus smal lestY ) 2) + 200 ) gotXandYPos = true

Cameraimg Save ( contImage png )

r e turn Cameraimg

76

• Acknowledgement
• Abstract
• Introduction
• • Motivation
  • Electromagnetic Systems and Magnetic Control
  • Thesis Structure
  • • Experiments preparation
    • • Modeling of a KUKA IIWA Serial-Link Manipulator
      • • Robots in Minimally Invasive Interventions
        • KUKA iiwa r800
        • Interface of the KUKA robot
        • • Communication protocols for connecting the robot server with a c application
          • • Additional features added to the robot arm to allow a free motion in the null frame and motion to a target position
            • • Gravity compensation feature of allowing robot arm motion in the null space
              • • Motion feature for driving the robot arm to an aimed position
                • • Camera calibration to extract the intrinsic parameters
                  • • Camera modelling
                    • • Feature Tracking of Microparticles
                      • Kinematic Modeling of the Robotic Arm and Microparticle
                      • Control of the Magnetic-Based Robotic System
                      • • Experiments
                        • • Position control of paramagnetic microparticles in xz-plane
                          • • Upward movement of the cluster
                            • Downward movement of the cluster
                            • • Position control of paramagnetic microparticles
                              • • Motion control trials
                                • • Control of paramagnetic microparticle along a trajectory
                                  • • Conclusions and Future work
                                    • • Conclusions
                                      • • Future work
                                        • • Bibliography
                                          • • List of Figures
                                            • List of Tables
                                            • Appendix
                                            • • Communication protocol User Datagram Protocol (UDP)
                                              • • Sending bytes message from robot server to C
                                                • • Receiving byte message from C application in the robot server
                                                  • • Creating a UDP communication in C application
                                                    • Recieving a byte message using udp in java and C
                                                    • Sending a byte message using udp in java and C
                                                    • Image processing application example