1
z
deti departamento de electrónica telecomunicações e informática universidade de aveiro
Introduction to VR and AR 3D Interfaces – Input
Paulo Dias
Outline
• Introduction to VR and AR
• 3D User interfaces – Input– Trackers
– Navigation and manipulation interfaces
– Gesture Interfaces
• Camera Calibration and Augmented Reality
• Practical labs: ARToolkit
2
Why Virtual Reality / 3D interfaces?
• 3D Digital information is common (3D reconstruction, games, CAD, Design,…)
• Interaction is still mainly 2D – keyboard and mouse from 70
• Virtual Reality: an attempt to go beyond 2D interaction and immerge user in 3D world
3
Apple II – from 1977
What is Virtual Reality
“A high-end user interface that involves real-time simulation and interaction through multiple sensorial channels.” (vision, sound, touch, smell, taste) (Burdea and Coiffet., 2003)
4
Virtual and Augmented Reality
5
Real
Environment
Augmented
Reality
Augmented Virtuality Virtual
Environment
Mixed Reality
Paul Milgram [Milgram et. al 1994]
Main challenges in VR/AR
• How to interact with users – Input Devices
• How to immerge user in the world – Output Devices
• =>No single solution: several trials with different setups
6
2
Virtual Reality example
• Equipment
– Tracker 3DOF (Yaw, Pitch, Roll)
– HMD SVGA estereoscópico
– WiiMote
– Wintracker 6DOF
• Graphical libraries
– VTK (Visualization Toolkit)
– OGRE
9
Interaction in RV
• Orientation– Yaw, Pitch e Roll from 3DOF tracker
– Direct mapping to camera
• Navigation– Acceleration sensor
– Movements map to actions
• Navigation mode– Axis, Point of view, etc…
10
Augmented Reality - equipment
• Equipment
– Virtual Reality + video camera
– Video see-through
• AR Toolkit library
– Marker detection
– Superposition of virtual elements in the world
• Virtual Environments in VRML
– Configuration file in txt
11
Results
12
Virtual and Augmented Reality Prototype
More recent setup
13
Input devices : Motion trackers
– Magnetic
– Mechanical
– Acoustic
– Optical
– Inertial
– Hybrids (combination of severals)
14
Polhemus Fast-Track
Boom
3D Mouse - Logitech
InertiaCube 3 - Intersense
Finger Tracking - ART
3
15
Input Devices Input devices
• Trackers:– Mechanical– Magnetic (AC, DC)– Optical– Ultrasonic– Inertial, – ...
• Navigation and manipulation interfaces:– Tracker-based– Trackballs– 3D mice, – ...
• Gesture interfaces:– Didjiglove– Cyberglove– ...
16
Virtual objects have 6 degreesof freedom (D.O.Fs):
-three translations;
-three rotations.
(Burdea and Coiffet., 2003)
Trackers measure the motion of “objects” (e.g. user head) in a fixed system of coordinates.
Tracker is a special purpose H/W to measure the real-time change in a 3D object position and orientation
Trackers
17
Without the head tracker - the image- the sound cannot change to match thehead posture
Required tracking accuracy: Image > sound
(Burdea and Coiffet., 2003)
Example magnetic in HMD
18
Body Tracking:
• Head
• Hand and fingers
• Torso
• Feet
• ...
Indirect tracking: Using physical objects (props and platforms)
Direct tracking: Tracking directly the body parts
What is usually tracked?
19
• Measurement rate – Readings/sec
• Sensing latency
• Sensor noise and drift
• Measurement accuracy (errors)
• Measurement repeatability
• Tethered or wireless
• Work envelope
• Sensing degradation
Trackers characteristics
20
4
• Accuracy
• Jitter
• Drift
• Latency
• Tracker update rate
Tracker performance parameters:
21
Tracker performance parameters should be analyzed to match a solution for sensorial channel and budget of an application
Real object position
Accuracy
Tracker position
measurements
Tracker characteristics
22
(Burdea and Coiffet., 2003)
Accuracy:Difference between the object’s actual 3D position and that reported by the measurement
Real object fixed
position
Signal noise
Time
Tracker data
Tracker characteristics
23
(Burdea and Coiffet., 2003)
Jitter:Change in tracker output when the tracked object is stationary
Real object fixed
position
Sensor drift
Time
Tracker data
Tracker characteristics
24
Drift: Steady increase in tracker error with time
(Burdea and Coiffet., 2003)
Real object
position
Sensor latency
Time
Tracker data
Tracker characteristics
25
Latency:Time delay between action and result: time between the change in object position/orientation and the time the sensor detects this change
(Burdea and Coiffet., 2003)
Tracker characteristics
26
(Burdea and Coiffet., 2003)
Tracker update rate:Number of measurements that the tracker reports every second
If the same tracker electronics is used to measure several objects, the sampling rate suffers due to multiplexing
Dedicated electronics to each tracked object
Same electronics to each tracked object
5
Mot used trackers
– Magnetic
– Mechanical
– Acoustic
– Optical
– Inertial
– Hybrids (combination of severals)
27
Polhemus Fast-Track
Boom
3D Mouse - Logitech
InertiaCube 3 - Intersense
Finger Tracking - ART
A mechanical tracker consists of a serial or parallel kinematic structure composed of links interconnected by sensorizedjoints.
Mechanical trackers
28
(Burdea and Coiffet., 2003)
Mechanical tracker - Push 1280 stereo display (Fakespace Inc)
Item is no longer available
Pros
• Use sensors imbedded in exoskeletons to measure position
• Have extremely low latencies
• Are immune to interference from magnetic fields and large
metal objects
Cons
• Limit the user’s freedom of motion
• Can be heavy if worn on the body
Mechanical trackers
29
Mechanical tracker - Example of an exoskeleton
30
http://www.youtube.com/watch?v=uJza6G-7tD4
A magnetic tracker is a non-contact position measurement device that uses a magnetic field produced by a stationary TRANSMITTER to determine the real-time position of a moving RECEIVER element
may be ACDC
31
Magnetic trackers
• Use low-frequency magnetic fields to measure position
• Fields are produced by a fixed source
• Size of source grows with the tracker work envelope
• The receiver is attached to the tracked object and has three perpendicular antennas
• Distance is inferred from the voltages induced in the antennas – needs calibration…
Magnetic trackers
32
6
Magnetic trackers with data glove
33
(Burdea and Coiffet., 2003)
Magnetic trackers widely used but expensive
34
Ascension
Tracking, guiding, and localizing medical instruments within a patient's body
3D measurement and analysis of human movement for biomechanical purposes: sports, performance, and design
Track head and objects for matching computer-generated imagery with head direction; weapon aiming and interactive instruction
Tracking of head, hands and 3D pointers for interaction with large scale and immersive displays
Wireless suit ascension technologies
35
Magnetic Tracker – Razor Hydra [low cost 79€]
36
• True six degree-of-freedom magnetic motion tracking
• Ultra precise sensor for 1mm and 1 degree tracking
• No line of sight to controllers required
• Low-power magnetic field
• low power consumption
• Low latency feedback
Razer Hydra
• Use mechanical measurements to reduce errors
• Sensor noise – variation in measurement with no
real object motion – solved by over-sampling
• Size of errors grow from source outwards
• Errors both in position and orientation
Magnetic trackers Calibration
38
Magnetic trackers accuracy degradation
39
(Burdea and Coiffet., 2003)
7
• DC trackers are immune to non-ferromagnetic metals
(brass, aluminum and stainless steel)
• Both DC and AC trackers are affected by the presence of ferromagnetic metals
(mild steel and ferrite)
• Both are affected by copper
• AC trackers have better resolution and accuracy
• AC trackers have slightly shorter range
DC vs AC magnetic trackers
40
A non-contact position measurement device that uses an ultrasonic signal produced by a stationary transmitter to determine the real-time position/ orientation of a moving receiver.
Ultrasonic trackers
41
(Burdea and Coiffet., 2003)
• Use low-frequency ultrasound to measure position
• Sound produced by a fixed triangular source (speakers)
• Number of sources grows with the tracker work envelope
• The receiver is triangular and attached to the tracked object and has
three microphones
• Distance is inferred from the sound time of flight
• Sensitive to air temperature and other noise sources
• Requires “direct line of sight”
• Slower than magnetic trackers (max 50 updates/sec)
Ultrasonic trackers
42
(Burdea and Coiffet., 2003)
http://inition.co.uk/3D-Technologies/logitech-head-tracker
6 DOFsUpdate rate - 50 / secLatency – 30 ms
Ultrasonic trackers - Logitech
A non-contact position measurement device that uses optical sensing to determine the real-time position/ orientation of an object
Optical trackers
44
(Burdea and Coiffet., 2003)
outside-looking-in inside-looking-out
LaserBIRD optical tracker
Optical trackers – outside looking in
45
(Burdea and Coiffet., 2003)
http://www.ascension-tech.com/realtime/laserbird2.php
Sub-degree and sub-millimeter accuracyMeasurement rate of 240 meas/secTracking response: 7.17 ms
8
Vicon MX
• Uses 4 Mpixel cameras with own 120 LED array (infrared, or visible red). Accuracy 0.02 of a pixel
• Camera has real-time onboard image processing (masking and thresholding)
• Resolution 2352x1728 @ 160 fps
• 8 cameras are connected to a MX net unit which then communicates with the PC
Optical trackers – outside looking in
47
(Burdea and Coiffet., 2003)
• MX Link connects several MX Net units for more cameras
• To interface with other devices like a force plate, sensing glove or eye tracker – use a MX control unit
• User wears reflective markers (small spheres)
(Burdea and Coiffet., 2003)
Optical trackers – outside looking in
• The best accuracy is close to the work envelope
• Very large tracking surface and resistance to visual occlusions (line of sight)
Inside-out optical tracker advantages/ disadvantages
49
(Burdea and Coiffet., 2003)
Inertial motion tracker
50
Xsens
Inertial trackers
• No interference from metallic objects
• No interference from magnetic fields
• Large-volume tracking
• “Source-less” orientation tracking
• Full-room tracking
Hybrid – Ultrasonic/Inertial trackers
51 52
Intersense IS 900
(Burdea and Coiffet., 2003)
Immune to metalic objects Accuracy: 2.0 - 3.0 mm Update Rate: 180 Hz Nominal
(120 Hz Nominal Wireless)Latency: 4 ms typical
http://www.intersense.com/pages/20/14
Inertial trackers
9
InterSense Stereo stylus tracker
Accelerometer
Ultrasonicemitter
53
http://www.mindflux.com.au/products/isense/is900.pdf
But…
• Accelerometer errors lead to decreased accuracy
• Errors grow with time!
• Gyroscope errors compound position errors
• Needs independent position estimation to reduce “drift”
54
Hybrid – Ultrasonic/Inertial trackers
InertiaCAMFixed-focus lens, on-board DSPWeight 35 gramsTracking range 10 m per 10 cm of fiducial diameter
Fiducials
• 6 DOF, accuracy 3 mm, 0.1 degree,
• Latency 2 ms;
• Requires a minimum of 1 fiducial
• for every 2 m2 at 2 m from camera;
• Update rate 120 Hz;
Hybrid vision - inertial
60
IS 1200
• Navigation interfaces allow relative position control of virtual objects
(including a virtual camera)
• Gesture interfaces allow dexterous control of virtual objects and interaction
through gesture recognition.
Navigation and Gesture Input Devices
61
• Trackballs
• 3D mice
• 3D probes
• Cubic Mouse
• Wiimote, …
• Perform relative position/velocity control of virtual objects
• Allow “fly-by” application by controlling a virtual camera
Navigation Input Devices
62 63
http://www.youtube.com/watch?v=1WuH7ezv_Gshttp://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.84.6127&rep=rep1&type=pdf
- Allows to intuitively specify 3D- coordinates
- The rods represent the X, Y, and Z axes of a given coordinate system
Cubic mouse
10
64
http://www.spacecontrol.us/3d-maeuse-spacecontroller-funktionsprinzip.htmlhttp://www.3dconnexion.com/products/what-is-a-3d-mouse.htmlhttp://www.youtube.com/3DMaus
• Sends relative coordinates
• Allows - to move 3D objects in a more intuitive way than a 2D mouse- allows rotation around each axis
• Price ~300 USD
3D mouse Space controller
http://www.axsotic.com/overview.html
- Has no mechanical sensors that can generate unwanted behaviors
- The specially developed polymer springbodies create a smoothly consistent workflow
3D-Spheric-Mouse
• Are sensing gloves such as:
- Fakespace Pinch Glove (switches)
- 5DT Data Glove (fiber-optic)
- DidjiGlove (capacitive)
- Immersion CyberGlove (stain gauges), …
• Have larger work envelope than trackballs/3-D probes
• Most need calibration for user’s hand
68
5DT Data Glove
DidjiGlove5DT Data Glove
CyberGlove
Pinch Glove
Gesture Input Devices
Finger Degrees of Freedom
69
70
(Burdea and Coiffet., 2003)
Hand work envelope vs. interface type
- No joint measures ; does not need user calibration
- Detects contact between any two fingers
- RS-232 protocol communicates - gesture information - delta time stamps
- Does not provide location in space
- A mount compatible with - Ascension - Polhemus(electromagnetic trackers)
71
The Pinch Glove (Fakespace Co.)
http://inition.co.uk/3D-Technologies/fakespace-labs-pinch-gloves
Pinch glove (Fakespace Co.)
11
The glove interface: A) five-sensor version; B) newer design
Price: ~900$
http://www.5dt.com/products/pdataglovemri.html
http://www.vrealities.com/products/data-gloves/5dt-data-glove-5-ultra-2
A) One optical fiber/finger Roll/pitch sensing 5DT Data Glove Ultra
100 datasets/sec, 12 bit A/D flexion resolution, wireless version transmits data at 30 m, needs calibration
73
5DT Data Glove (5Th Dimension Technologies)
75
(Burdea and Coiffet., 2003)
http://www.youtube.com/watch?v=RqOcuzRP1aQ#t=21
5DT Data Glove (5Th Dimension Technologies)
78
http://inition.co.uk/3D-Technologies/didjiglove
• Captures the position of fingers by means of 10 capacitive bend sensors
• Uses one flex sensor/finger
• Angles are obtained by measuring voltages
• Communicates via RS232
DidjigloveThe DG5 glove
• 100 gestures/sec
• Small latency (10 ms)
• Sensor resolution 10 bit (1024 points)
• Cannot measure individual joints
• Needs user specific calibration
• Less expensive (about $500)
http://www.vrealities.com/products/data-gloves/dg5-vhand-glove-3-0
79
• Uses 18-22 linear sensors – electrical strain gauges
• 112 gestures/sec “filtered”
• Angles are obtained by measuring voltages on a Wheatstone bridge
• Sensor resolution 0.5 degrees, but errors accumulate to the fingertip
• Sensor repeatability 1 degree
• Needs calibration when put on the hand;
• Expensive (about $10,000)
81
http://www.metamotion.com/hardware/motion-capture-hardware-gloves-Cybergloves.htm
http://www.cyberglovesystems.com/
The CyberGlove (Cyber Glove Systems)
82
http://www.cyberglovesystems.com/
The CyberGlove (Cyber Glove Systems)
12
Specifications Pinch Glove
5DT Data Glove DG5 Glove
DG5 VHand
CyberGlove
Number of
sensors
Sensor type
Record/sec
Sensor
resolution
Communication
rate
Wrist sensors
7/glove
(2 gloves)
Electrical
NA
1bit
(2 Points)
Wired
(19.2kb)
None
5 or 14/glove
(1 glove)
Fiber-optic
100(5DT 5W),
200(5DT 5)
8bit
(256 Points)
Wireless(9.600kb)
Wired(19.2kb)
Pitch
(5DT 5 model)
5 /glove
(1 gloves)
?(ink film)
100
25 (DG 5 VHand)
10 bit
(1024 points)
Wired (19.2kb)
Wireless (DG5 VHand)
Accelerometers
(DG5 Vhand)
18 or 22/glove
(1 glove)
Strain gauge
150(unfiltered)
112(filtered)
0.5°
Wired
(115kb)
Pitch and yaw
84
Low cost - Entertainment Driven
87
• Wii
• Kinect
• Playstation eye
• Leap
Wii
• Button
• Accelerometer
• Infrared Camera
• Feedback
– Visual
– Touch
– Sound
• Battery
• Communication
88
Playstation eye
• Optical tracking
89
Kinect – Structured Light
• Active – Infrared pattern
90
Main bibliography
- G. Burdea and P. Coiffet, Virtual Reality Technology, 2nd ed. Jonh Wiley and Sons, 2003
- Craig, A., Sherman, W., Will, J., Developing Virtual Reality Applications: Foundations of Effective Design, Morgan Kaufmann, 2009
-
- J. Vince, Introduction to Virtual Reality, Springer, 2004
92
13
Aknowledgement
The author of these slides is greteful to:
• Burdea and Coiffet for making available the slides supporting their book
• All colleagues and students that have contributed in any way
93