1
A Hierarchical Design Scheme for Application of Augmented Reality
in a Telerobotic Stereo-Vision System
BySyed Mohammed Shamsul Islam
Lecturer B and PhD studentDepartment of Computer Engineering
KFUPM
Co-authors: Prof. Mayez Al-Mouhamed Dr. S. M. Buhari Dr. Talal Al Kharobi
2
Outline
Introduction Design Issues Implementation Performance Evaluation Conclusions & Recommendations
3
Telerobotics:
A scheme that allows humans to extend their manipulative skills over a network.
Robot
LAN
Robot Contoller
Digital Cameras
`
`
`
Force feedback
Visual feedback
Visual feedback
Force feedback
4
Telerobotic Applications Hazardous situations
e.g. bomb disposal
Scaled-down and scaled-up situations e.g. micro-surgery
Situations affected by human presence e.g. closed-chest heart bypass
Teaching, training, maintenance, entertainment.
5
Problem of Conventional Tele-robotics
Time delay: Processing, copying video data, network delays Move-n-Wait strategy
Operator has to follow trial and error method to perform a task, which might be dangerous for example in telesurgery.
6
Improving Performance by Using AR
By overlaying virtual image with the real image operator e.g. surgeon can make a plan before going for actual teleoperation safety.
Rehearsal and correction can be made in the simulation plan.
Saving bandwidth by sending (less frequently) only the planned data.
7
Challenges of AR Graphical robot arm must mach
with real robot arm in the video.
Real video image
Graphics Subsystem
Graphic Co-ordinate
Space
RemoteCo-ordinate
Space(Video)
Bi-directional one-to-one mapping between graphic and real co-ordinate spaces
8
Challenges of AR
Camera Calibration: Finding accurate
camera calibration is challenging.
Viewpoint Registration
9
Design Issues
10
Telerobotic Stereo-Vision System to be Augmented
LAN
Vision Client
Decision Server Interface
Client User Interface
MasterArm Component
Master
Arm
Stereo Display
Eye-Shuttering Glasses
Robot
VisionServer
Internet
DecisionServer
Server UI
PUMAComponent
UnimationController
Force Component
DigitalCameras Horizontal disparity = 6 cm
N
11
Design MethodologyPerformance evaluation of the approach
Superimposition of the virtual object on to the stereo-video and augmented tele-manipulation
Interfacing to the telerobotic stereo-vision system
Moving capability of the virtual objects
Displaying the virtual objects
Drawing the virtual objects
Deriving the robot model equations
Evaluation
Augmentation ofStereo-vision
Interface
Animation
Virtual object rendering
Virtual object drawing
Mathematical models
12
Mathematical Model of the Robot Manipulator
PUMA 560 Slave Arm: 6 DOF 6 links are interconnected serially to each other except first and last. All the joints are rotational.
A rotational joint
13
Mathematical Model of the Robot Manipulator
Each link of the arm is attached with a frame of reference Ri (xi,yi,zi) to specify its position and orientation.
Link Li+1 rotates w.r.t Li when frame Ri+1 rotates w.r.t either axes of xi, yi, or zi
Link vector OiOi+1,i can be expressed as
1,11
,1
iiiiiiii OOMOO
],,[ ,1,1,11
iiiiiiii zyxM Where,
Now position and orientation of Oi+1 w.r.t Oi-1 can be expressed as
1,1111,11,11 iiiiiiiiiii OOMOOOO
11
11
i
iii
ii MMM
14
Mathematical Model of the Robot Arm
This model will give us a skeleton of the graphical arm.
Where,nn
n MMMMM 132
21
100 ......
nnnn
nn OOMOOOO ,100,100,0
)}(),({)( 00,0 nn MOOG
Position and orientation of the end effector is expressed relative to the frame of the base. So the geometric model of robot arm can be expressed as:
15
Building Body Shapes Around the Skeleton of Graphical Arm
Alternatives: Taking as is: link 3 and 4 are
trapezoidal, others are cylindrical.
Considering all as cylindrical
16
Building Body Shapes Around the Skeleton of Graphical Arm
Cylinder is drawn with finite element representation.
Increasing number of segments will improve quality of view but increase computational complexity.
Alternatives of primitive for drawing a cylinder:
Triangles: Triangle stripe, Triangle fan, Triangle list
Lines Line stripe, Line list v 1
v 2
v 3
v 4
v 5
v 6
v 7
(a ) Tr ia n g le s tr ip
17
Data Structure Design Types of data and nature of manipulation:
A link is represented by its start and end point co-ordinates, radius, axis of rotation, joint angle, orientation matrix etc.
Each link is also associated with the vertices of its body shape.
Links are serially connected and position and orientation of a link depends on position and orientation of its previous link.
There is frequent movement of links in teleoperation and each movement involves matrix calculation for new position and orientation of the links to be changed.
Objectives: To reduce processing time by reducing computation. To provide flexibility and generality
18
Data Structure Design
Our strategy: Each link is defined as an object
geometric data of each link is now apart from the other link,.
Configuration data (number of links, number of segments in cylinder etc) kept in separate data file.
World and view transformation matrices are combined to reduce matrix multiplication.
Vertex data of body shape of each link is stored in separate vertex buffer.
19
Displaying the Graphical Arm
Given the robot structure in its model space Rendering on the display screen:
Scene layout setup: Transformation matirces, Camera position, target position,
viewing angle, lighting. Viewing Models
Solid Model or Wire-frame Model Psychological studies and one performance studies in
telerobotic system of shows no significant diff. Wire-frame models have some adv.-reduce occlusion,
lesser rendering time. Rendering:
Scanline rendering, Z-buffering algorithm for HSR.
20
Sample Display of Graphical Arm
Solid viewWire-frame view
21
Movement of Graphical Arm in Joint Space
C al c ul ate ne w val ue s o f the jo i nt ang l e ,
Star t
C o m pute the P o s i t i o n ve c to r s
C o m pute the O r i e ntat i o n m atr i c e s
C o m pute the ve r t i c e s o f the bo dy shape s
R e nde r the w ho l e r o bo t w i th update d ve r t i c e s ( thus m o vi ng to a ne w
l o c at i o n)
N o
Ye s
R e c e i ve the i nc r e m e nt o f jo i nt ang l e ,
Ang l e w i thi n bo und?
).( 11
00ii
ii MMM
).( ,100,100,0 iiii
ii OOMOOOO
22
Movement of Graphical Arm in Cartesian Space
Start
Receive the incremental position (X) and orientation (M) matrices
Calculate new position, Xnew(t) and orientation, Mnew(t) matrices
based on current frame of reference
Calculate corresponding angular values Using Inverse Geometric Transformation
(new=G-1(Xnew, Mnew)
Angle within bound?
B
23
Movement of Graphical Arm in Cartesian Space…
Compute the vertices of the body shapes
Render the w hole robot w ith updated vertices
Compute the position vector of a ll the links
Compute the orienta tion matrices).( 1
100
ii
ii MMM
).( ,100,100,0 iiii
ii OOMOOOO
End
B
24
Acquisition of Real Video Image
The MDTF Approach is used for stereovision. It is multi-threaded, distributed and provide better performance in
video transfer. Real video image of the slave robot at the server side is
captured simultaneously by two video cameras. Then a reliable client-server connection is established and upon
a request from the client a stereo frame comprising of two pictures is sent over LAN through window sockets.
On the client side after detecting and making connection with the server pictures are received and displayed on the screen and on the HMD when connected.
A double buffer, concurrent transfer approach is used to maximize overlapped transfer activities between cameras, processor and the network.
25
3D Visualization Tools and Techniques
Technique Alternatives: Sync-Doubling, Page-flipping Chosen: Page-flipping
Provides higher resolution and avoid ”flashing” problem of 3D imaging.
Display device Alternatives: HMD, monitor, eye-shuttering glass Chosen: HMD, monitor
No flickering, easy to work
26
3D Visualization Example
A snap shot of a stereo-image.
27
Camera Calibration
Heikkila’s calibration method which provides very accurate results. It shows accuracy up to 1/50 of the pixel size.
Pinhole camera model with perspective projection and least-square error optimization.
The intrinsic camera parameters: Focal length, fc Principal point, cc Skew co-efficient, alpha_c Lens distortion co-efficient, kc
28
Camera Calibration…The aspect ratio= fc(2) / fc(1)
The field of view angle can be calculated from,
)2/cot(2
)2(
)2/cot(2
)1(
y
x
fovW
fc
fovH
fc
29
Camera Calibration…
Extrinsic ParametersIf XX=[X,Y,Z] and XXc are co-ordinate of P in grid and camera ref, then
1_*1_ TcXXRcXX C
Where, translation vector Tc_1 is the co-ordinate vector of O in camera ref frame and the rotation matrix Rc_1 is the surface normal vector of the grid plane in the camera ref frame.
30
3D Visualization Algorithm
Step 1: Acquire video image and copy this memory stream to a surface( say FrontSurf).
Step 2: Copy the FrontSurf surface to another temporary surface (say, AugSurf).
Step 3: Draw graphical objects/change in graphical objects on AugSurf.
Step 4: Copy AugSurf to the surface that will be used to display (say, backSurf).
Step 5: Display backSurf with left camera image to left view port/monitor.
Step 6: Display backSurf with right image to right view port/monitor.
Step 7: Observe 3D with HMD.
31
Graphical Tele-manipulation
How Graphic Manipulation Corresponds to the Real If the base link of the real robot remains fixed relative to the
video cameras, the base link of the graphical arm will also remain fixed relative to the graphical cameras.
The end-effector of the graphical arm can be manipulated in the graphical coordinate space, relative to objects in the task space (keeping base link in same location of real robot base).
32
GUI Design
Client Side Input/Output Specification Connecting to the server PC, master arm and HMD Receiving and displaying the stereo video. Taking user's input for movement of the real and graphical
robot for simulation
Means of User Interaction: Master arm Joystick Keypad Mouse
33
Implementation
34
Tools Used
Client-server communication platform MS .NET with .NET Remoting
Programming Language Visual C#.NET
Graphics Tool Microsoft DirectX
3D Graphics API Alternatives:
Windows GDI Java3D by Sun MicroSystems Open GL (Open Graphics Language) by SGI Silicon Graphics Direct3D
35
Motivation for Using Direct3D It increases performance by using hardware acceleration.
It allows applications to run full-screen instead of embedded in a window. As we are using HMD this feature is very helpful in our case to have the 3D effect.
Direct3D perfectly match with Microsoft's various Windows operating systems and with Microsoft .NET framework which are used in our telerobotic client-server framework
36
Direct3D Architecture
W indows Application
Direct3D API
G DI
Hardware Abstraction Layer(HAL)
Device Driver In terface(DDI)
G raphics Device
37
Graphics Implementation
Virtual Object Modeling Module Defines the structure of the virtual objects (robot, cubes
etc.) Handles functions like movement of virtual objects.
Display Module: Synchronization of real and virtual data Projection on video surface Augmentation of real video Page Flipping for HMD stereo visualization
38
Interfacing to Telerobotic Stereovision System Client Modules:
LAN
VisionClientComponent
DXInterface Component
Display Surface(HMD, Monitor
etc)
Stereo Video
Camera Calibration Component
Left, Right Images
DecisionServer
MasterArmComponent
),( MX
Robot Model
Augmented Video
),( newnewMX
),( MX
Robot Client
Virtual object
Modeling
User Interface
39
GUI Implementation Main Form
40
GUI Implementation Stereo Form
41
Performance Evaluation
42
Evaluation of Graphical System: Refresh Rate:
Environment Graphics Complexities Avg. Refresh Rate
(frame/sec)
Graphics without overlaying real video
image
Without any drawing 273.36
With an object and the graphical arm in the scene 243.74
Only graphical arm with 8 segments in each cylinder 253.59
Only graphical arm with 50 segments in each cylinder (solid view)
239.78
Graphics without overlaying real video
image
Without any drawing 11.498
With an object and the graphical arm in the scene 11.384
Only graphical arm with 8 segments in each cylinder 11.347
Only graphical arm with 50 segments in each cylinder (solid view)
11.325
[Average is taken over 1000 samples, running on Pentium-4, 2GHz machine with 1GB RAM]
43
Time Required for Rendering Graphical Arm
Environment Graphics Complexities Rendering Time (ms)
Wire-frame
Solid
Graphics without
overlaying real video image
Only graphical arm with 8 segments in each cylinder
64.088 64.283
Only graphical arm with 50segments in each cylinder
65.034 65.151
Graphics overlaying real
video image
Only graphical arm with 8 segments in each cylinder
105.02 108.037
Only graphical arm with 50segments in each cylinder
105.37 109.362
[Average is taken over 1000 samples, running on Pentium-4, 2GHz machine with 1GB RAM]
44
Accuracy Re-projection error with the calibration method: [Pixel Error (0.11689,0.11500)]
Pix
el A
xis
in t
he y
dire
ctio
n
Pixel Axis in the x direction
45
Comparison to Other Apporaches Iqbal, A. [1] augmented with only a small red ball at the
position of gripper in comparison to our whole graphical arm. Iqbal, A. [1] used Faugeras [4] calibration with Kuno[5]’s affine
frame of reference which led him to noticeable mismatch with real error in the matching shown in his figure. Whereas our computer vision-based calibration reduces error upto 1/50 of pixel size.
J. Vallino reports in his PhD thesis refresh rate of 10fps to be required for AR, when we gets above 11-17fps after overlaying graphical arm with live stereo video.
Graphics rendering of our system is faster than Iqbal, A. [1],[2][3] for our use of Direct3D.
Our system is comparatively cheaper due to the use of commodity hardware (PC) and software.
46
Conclusion
47
Summary of the Work and Contributions Using hardware accelerated graphics rendition that
provides us with excellent refresh rate of the output screen.
Improvement in accuracy of execution by using better calibration method and graphic aids (accuracy up to 1/50 of pixel size).
User-friendly graphical user interface for simple manipulation in the telerobotic AR system.
48
Summary of the Work and Contributions…
Flexible and generalized data structure suitable for telerobotic visualization.
Identifying design strategy for intelligent switching in VR and AR mode for ensuring QoS.
Use of cheap and commercially available hardware and software
Can be used as a cheap and flexible visual tool for showing robot manipulation in the classrooms.
49
Future Research Directions Providing an intelligent system to switch
between VR and AR modes of operation based on network delays to ensure QoS.
Using multi-processor system for processing video and graphics data more efficiently.
Using commercial software available to extract exact 3D model of the workspace objects which will facilitate more accurate task manipulation.
50
References
1. Iqbal, A. Multistream realtime control of a distributed telerobotic system. M.Sc. Thesis, King Fahd University of Petroeum and Minerals,June 2003.
2. A Rastogi, P. Milgram, and D. Drascic. Telerobotic control with stereoscopic augmented reality. SPIE, Vol.2653: Stereoscopic Displays and Virtual Reality Systems III:135{146, Feb. 1996.
3. R. Marin, P.J. Sanz, and J.S. Sanchez. A very high level interface to teleoperate a robot via web including augmented reality. Proc. IEEEInternational Conference on Robotics and Automation, 2002 ICRA '02, Vol.3:2725 - 2730, May 2002.
4. O.D. Faugeras and G. Toscani. The calibration problem for stereo. Proceedings of Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, Vol. 5, No. 3, June, 15-20 1986.
51
References
[5]Y. Kuno, K. Hayashi, K.H. Jo, and Y. Shirai. Human-robot interface using uncalibrated stereo vision. International Conference on Intelligent Robots and Systems 95, 1:525{530, 1995.
[6]J. Abdullah and K. Martinez. Camera self-calibration for the ar-toolkit. The First IEEE International Workshop on Augmented Reality Toolkit, page 5, Sept. 2002.
[7]J.-Y. Herve, C. Duchesne, and V. Pradines. Dynamic registration for augmented reality in telerobotics applications. IEEE International Conference on Systems, Man, and Cyberneticsn, Vol.2:1348-1353, October 2000.
52
