Post on 21-Aug-2020
transcript
Nicolas D. Georganas
Collaborative Virtual Environments
Nicolas D. Georganas, FRSC, FIEEECanada Research Chair in Information Technology
Distributed and Collaborative Virtual Environments Research Laboratory (DISCOVER)
School of Information Technology and EngineeringUniversity of Ottawa
georganas@discover.uottawa.cawww.discover.uottawa.ca
Nicolas D. Georganas
Presentation
• Virtual Reality & Collaborative Virtual Environments
• Applications• Standards• Performance Issues
Nicolas D. Georganas
Virtual Reality• Virtual Reality or Real Virtuality?• Originally, the term referred to 'Immersive
Virtual Reality.' – the user becomes fully immersed in an
artificial, 3-D world that is completely generated by a computer.
• The head-mounted display (HMD) was the first device providing its wearer with an immersive experience (1965)
Nicolas D. Georganas
Virtual Reality
• Desktop VR– Two dimensional medium , used for 3-D
• Augmented Reality– the ability to see the real world and the virtual
world at the same time. • Immersive VR
– the user becomes fully immersed in an artificial, three-dimensional world that is completely generated by a computer
Nicolas D. Georganas
CAVE
Nicolas D. Georganas
DISCOVER Laboratory @ SITE: Distributed and Collaborative Virtual
Environments ResearchDISCOVER Laboratory ($4 million):
ONYX 3400 with 3 graphic pipes and 12 processorsMechdyne FLEX 3-panel screen$1 million of other VR and Haptic equipment
Professors:N.D.Georganas (Distributed Virtual Environments,
Augmented Reality, Tele-Haptics, Real-Time HLA)E.M.Petriu (Virtual Prototyping, Virtual Theater,
Tele-Haptics, Intelligent Sensors)E.Dubois (Virtualized Environments)A.El Saddik (Tele-Collaboration, Web Engineering)W. Lee (Human Animation)S. Shirmohammadi (Tele-Collaboration)
Nicolas D. Georganas
DISCOVER Flex CAVE
Nicolas D. Georganas
Zaxel Virtual Viewpoint
-12 camera capture and avatar/object production
Nicolas D. Georganas
DISCOVER CAVE Stations
Nicolas D. Georganas
DISCOVER CAVE Stations
Nicolas D. Georganas
Haptics
• Hapicts: derived from the Greek verb “haptesthai” meaning “to touch”
• Haptics introduces the sense of force and touch in human-computer interaction
• Tele-haptics can be defined as the use of haptics in a distributed network context
Nicolas D. Georganas
DISCOVER Haptic Devices
• SensAble Phantom Desktop
6 DOF Input. 3 DOF output.High-Fidelity force feedbackDirect parallel port connectionVery impressive
Nicolas D. Georganas
DISCOVER Haptic Devices (cont.)
Nicolas D. Georganas
DISCOVER Haptic Devices (cont.)• Four MPB Freedom
6S6 DOFHigh-Fidelity force feedbackConnects through HRTCC box to your computer
Nicolas D. Georganas
DISCOVER Haptic Devices (cont.)
Immersion CyberForce/ CyberGrasp/ CyberGlove
Nicolas D. Georganas
DISCOVER Haptic Devices (cont.)HAPTIC MASTER
Nicolas D. Georganas
• a simulated world runs not on one computer system, but on several.
• The computers are connected over a network (possibly the global Internet) and people using those computers are able to interact in real time, sharing the same virtual world.
• On each host there are a number of "entities"(things in the virtual environment) that communicate their changing state by sending "update messages".
Distributed Virtual Environments
Nicolas D. Georganas
The Origin of Distributed Virtual Environments
• Department of Defense Networked Virtual Environments – SIMNET – Distributed Interactive
SimulationPromote IEEE standard
• Networked Games and Demos – SGI Flight and Dogfight – Doom – Other GamesInteractive graphics applications
Nicolas D. Georganas
Distributed Virtual Environment
---------------------------------------------
--------------------------------------
-------
CVE
Real-time Interaction-Between the customers-Between customers and salesmen-Between customers and goods
Shared Environment-Global coordinate system-Static background-Dynamic entities
Mutual awareness-Customers can “see” and “talk” to each other in the CVE
--------------------------------------
-------
Server
Nicolas D. Georganas
Collaborative Virtual Environments (CVE)
• Distributed Virtual Worlds• Users represented there by their “avatars”
(=incarnations)• Collaborative work performed in CVE• A Collaborative Virtual Environment
requires an unconventionally broad range of networking, database and graphics capabilities.
Nicolas D. Georganas
CVE Data Types• real-time audio and video data: data carrying the
video or audio streams in the environment.• object/scene description data: data describing an
object and its attributes; e.g., description of an avatar.• CSCW data: this represents traditional 2D
collaborative data such as whiteboards.• control data: used by the system to perform tasks
such as consistency control, management, and so on.
• update messages: such as avatar/vehicle motions.• synchronous collaborative update messages:
update messages representing a synchronous collaborative task.
Nicolas D. Georganas
CVE Issues
• human-factors research in VR has traditionally focused on the development of natural interfaces for manipulating virtual objects and traversing virtual landscapes.
• CVEs require the consideration of – how participants should interact with each
other in a shared space, in addition to how co-manipulated objects should behave.
Nicolas D. Georganas
CVE issues• Other issues include:
– how participants should be represented in the collaborative environment;
– how to effectively transmit non-verbal cues that real-world collaborators so casually and effectively use;
– how to best transmit video and audio via a channel that allows both public addressing as well as private conversations to occur; and
– how to sustain a virtual environment even when all its participants have left.
Nicolas D. Georganas
Causality in CVE
Time
Simulator A(tank)
Simulator B(target)
“fire”
“targetdestroyed”
Simulator C (observer)
eventmessage
real world simulated world
the observer should see the tank fire before the target is destroyed.
• “Things” happen in the real world in a certain order (e.g., cause & effect).• It should appear that events in the simulated world happen in the same order as the real world actions that they represent.• If the message for the “fire” event is delayed in the network, the observer will “see” the target is destroyed before the tank fired upon it!
Nicolas D. Georganas
Presentation
• Virtual Reality & Collaborative Virtual Environments
• Applications• Standards• Performance Issues
Nicolas D. Georganas
Are needed in order to better train medical students and doctors alike;
Allow errors to be made on virtual rather than real patients;Allow rare cases to be modeled and studied;Alleviate the animal rights controversy and the shortage of
cadavers problem;Permit more enjoyable and efficacious treatment for patients
(example rehab);Allow more realistic examination (board exams);Allow reduction in medical costs (less liability suits, cheaper
training – emergency response).
MEDICAL APPLICATIONS OF VRMEDICAL APPLICATIONS OF VR
Nicolas D. Georganas
DISCOVER Tele-Surgery (Trachea) Training
Nicolas D. Georganas
DISCOVER Tele-Surgery (Trachea) Training
Nicolas D. Georganas
DISCOVER Tele-Surgery (Trachea) Training
Nicolas D. Georganas
DISCOVER Tele-Surgery (Trachea) Training
Nicolas D. Georganas
Virtual Environment Technology Laboratory, Univ. of Houston and
NASA Johnson Space Centre, US
• Shared Virtual Environments for Training http://www.vetl.uh.edu/overview/overview2.html
Nicolas D. Georganas
Human Interface Technology Laboratory (HITL), Univ. of Washington, US
• Shared Space: Collaborative Augmented Reality• http://www.hitl.washington.edu/projects/shared_space/
Nicolas D. Georganas
Human Interface Technology Laboratory (HITL), Univ. of Washington, US
•collaborative Web browser in which users can see each other and virtual Web pages simultaneously. Using a see-through display, participants are naturally aware of each other's facial expressions, body language, gestures and gaze.
Nicolas D. Georganas
Sony Computer Science Lab., Sony Co. , Japan
TransVision: A Collaborative Augmented Reality Testbed
http://www.csl.sony.co.jp/person/rekimoto/transvision.html
•Augmented reality for collaborative designing
•The system uses the palmtop video-see-through display instead of bulky head-mounted displays. The user can see a computer-generated 3D model superimposed on the real world view
Nicolas D. Georganas
SICS - Swedish Institute of Computer Science, Sweden
Multi-user Virtual Workplace Conferencing
Nicolas D. Georganas
Virtual Theater projectDISCOVER Lab, UofO
VisualFeedback(s)
Machine/ Human Interface
TactileFeedback(s)
ForceFeedback(s)
VideoSensor(s)
TactileSensor(s)
ForceSensor(s)
StructuredLight
Human/Machine Interface
Object Recognition
Motion TrackingVirtual Object Manipulation
Object Shape &Behavior ModelsObject Interaction Models
Computer Generated Models Animation Script
Pseudo-Random Encoding
VIRTUAL SCENE / THEATER (where virtual objects interact)
Head-Tracking system
Body Tracking System
Nicolas D. Georganas
VIRTUAL THEATER for Industrial Training: A Collaborative Virtual
EnvironmentDISCOVER Lab , Univ. of Ottawa
Nicolas D. Georganas
CVE for Industrial Training• Reduces Training Expenses
– Waste of equipment which likely gets damaged– Maintenance of Training Facilities– Training of users in remote locations
• Very little cost associated with creating virtual objects– It can be replicated as many times as necessary
• Any damage done to the object does not incur in financial loss
• Maintenance fee is very low• Actual objects may be used for its intended use
Nicolas D. Georganas
Industrial Training Prototype
Nicolas D. Georganas
Head Tracking
• Head tracking allows natural way of controlling the avatar’s head.
• Users have a better awareness of others while listening to their voice and watching their avatar’s animation
• Extension to body motion detection allows Virtual Theater like collaboration
• 3D Scene is projected in a wall for stereoscopic view through 3D shutter glasses and more natural experience for the users.
Nicolas D. Georganas
V-COM: CVE in e-CommerceDISCOVER Lab, UofO
User ProfileManager
User
VirtualWorld
RTI VirtualWorld
VirtualShopping
Mall
RTI VirtualShopping Mall
Security Manager
QoS Broker
Sales Agent
CGIRegistration/Login
Negotiation
Search Agent
CGIQuery
Transactions
Multimedia catalogs
HTTP Server HTTP Server
Vendors
Nicolas D. Georganas
User Interface
Nicolas D. Georganas
Collaborative e-Commerce
federate Afederate B
HLA MiddlewareTCP/UDP/IP Network
federate C
federate A
Nicolas D. Georganas
Mobile Collaborative Augmented Reality
DISCOVER Lab, UofO
• Computer Supported Collaborative Work(CSCW) – Allows a computer to be used as a medium
for human communication • Mobile Collaborative Augmented Reality
– The combination of augmented reality, mobile computing and CSCW promises exciting applications for industrial training
Nicolas D. Georganas
Augmented Reality (AR) Prototype
A trainee works with a wearable computer
Where and which virtual objects should be superimposed on the real world?
A marker with a square region and some pattern inside of the square region is used
Supports multiple trainees and trainers distributed in different locations
Nicolas D. Georganas
AR Prototype (cont’)
(a) A physical board with virtual chips around
(c) A virtual chip is laid on the board
(d) The virtual chip is rotated by remote trainee B
(b) Local trainee A is moving a virtual chip
Nicolas D. Georganas
Presentation
• Virtual Reality & Collaborative Virtual Environments
• Applications• Standards• Performance Issues
Nicolas D. Georganas
VR Standards: Rendering• OpenGL
– Introduced by SGI in 1992– Multiplatform– Hardware acceleration
• DirectX– Created by Microsoft– Supports Win9x, WinNT (DirectX 3) and Win2000– Hardware acceleration
• VRML– VRML 1.0 based on SGI’s Open Inventor– VRML 2.0 in 1996– ISO standard in 1997– Browsers based on OpenGL or DirectX
• Java3D– Introduced by Javasoft in 1998– Multiplatform– Build on top of OpenGL and ActiveX
Nicolas D. Georganas
VRML BrowsersVRML-Browsers are software tools that generate views from a VRML file.
Nicolas D. Georganas
•VRML is a file format •It enables authors to control shapes and some animation and interaction •For more control, authors may write short programs in Java or JavaScript •To gain greater control, something more is needed...
•Java3D is a Java Application Programmer's Interface (API) •Java3D is a set of standardized classes for use by Java programmers •It enables authors to control shapes, animation, and interaction, PLUS rendering,
input devices, and lots more •It is (roughly) a Java-based superset of VRML
Nicolas D. Georganas
CVE Standards: Middleware• ALSP and DIS• Living Worlds (LW)
– Designed by the VRML Consortium– Aims at providing collaboration features for VRML– VRML 3.0 = VRML 2.0 + Living Worlds
• Open Community (OC)– Proposed by Mitsubishi– SPLINE (Scalable Platform for Large Interactive Networked
Environments) is a OC compliant implementation– SPLINE uses ISTP (Interactive Sharing Transfer Protocol)
• High Level Architecture (HLA)– Developed by the U.S. Defence Modeling Simulation Office (DMSO)– RTI (Run Time Infrastructure) is an HLA compliant implementation– HLA is now an IEEE standard (IEEE 1516)
• Java Shared Data Toolkit (JSDT)– Part of Java Media API developed by Javasoft– Java provision for real-time sharing of applets and/or applications
Nicolas D. Georganas
A History of DVEs1980
1985
1990
1995
SIMNET Start (83)
SGI Flight (84)SGI Dogfight (85)
Amaze (84)
SIMNET First Demo (86)NPS FOG-M (86)NPS VEH (87)NPS MPS-1 (88)
SIMNET to Army (90)NPSNET-1 (90)
BrickNet (91)
DIS (93)
DIVE (92)NPS-Stealth (93)
NPSNET-IV (93)Doom (93)Paradise (93)
HLA (97)
Nicolas D. Georganas
SIMNET
• SIMNET (simulator networking) is a distributed military virtual environment.
• SIMNET was begun in 1983 and delivered to the U.S. Army at the end of March 1990.
Nicolas D. Georganas
Distributed Interactive Simulation (DIS) protocol
• DIS (IEEE 1278), and its predecessor SIMNET, arose from real-time interactive computing.
• DIS is a government/industry initiative to define an infrastructure for linking simulations of various types at multiple locations to create realistic, complex, virtual worlds for the simulation of highly interactive activities.
• DIS is a virtual environment within which humans may interact through simulation at multiple sites that are networked using compliant architecture, modeling, protocols, standards and databases.
• DIS exercises involve the interconnection of a number of simulators in which simulated entities are able to interact within a computer generated environment.
• The simulators may be present in one location or be distributed geographically and the communications are conducted over the network.
Nicolas D. Georganas
Distributed Interactive Simulation (DIS)
The DIS (IEEE 1278) standard.Defines 27 different PDUs, only 4 of which (Entity State, Fire, Detonation, and Collision) are used by nodes to interact with the virtual environment.
A fully distributed, heterogeneous network software architecture.
Nicolas D. Georganas
LiveParticipants
SupportUtilities
Interface
Interfaces to Live Players
Simulations
Runtime InfrastructureFederation Management Declaration ManagementObject Management Ownership ManagementTime Management Data Distribution Management
High Level Architecture (HLA)
• OMG, IEEE, DoD standard
Nicolas D. Georganas
Federate FederateAmbassador
RTI Ambassador
Federate FederateAmbassador
RTI Ambassador
Components in a Federation Execution
RTI Executive( rtiexec.exe )
Federation Executive( fedexec.exe )
FEDFileFEDFile
Network
Inter-Process Communications
Federate FederateAmbassador
RTI Ambassador
Network
Inter-Process Communications
RTI Executive Federation Executive FEDFileFEDFile
Nicolas D. Georganas
HLA Services
Federation Management
Declaration Management
Object Management
Ownership Management
Time Management
Data Distribution Management
Nicolas D. Georganas
Example of Distributed Virtual Environment over HLA
federate Afederate B
HLA MiddlewareTCP/UDP/IP Network
federate C
federate A
Nicolas D. Georganas
Open CommunitySPLINE
Nicolas D. Georganas
Open Community, SPLINE
• SPLINE (Scalable Platform for Large Interactive Networked Environments) is a software library compliant with O.C. which provides an API for C (and promises of Java soon).
• ISTP (Interactive Sharing Transfer Protocol) is a hybrid protocol used by SPLINE which supports the sharing of information about a Virtual World among a group of user processes.
• OC (Open Community) is a proposed standard which provides many essential services necessary to make real-time multi-user cooperative environments possible.
Nicolas D. Georganas
OC (SPLINE) Functionality• Network Communication issues
– ISTP• Real-time audio transport ( no video)• Application neutral transport of large or
complicated objects– VRML models, sound files, etc.
• Filtering of Regions of Interest• Scalability to large Virtual World and
large number of users allowing users in not so powerful stations to participate.
Nicolas D. Georganas
OC World Model - High Level Overview
Nicolas D. Georganas
SPLINE: Scalability• World is partitioned in regions called locales.
– Each process is supposed to render a reduced number of locales (1- It’s own locale, 2- Neighbour locales)
• Each locale has its own multicast addresses– Each process only deals with relevant messages/audio
• Each locale has its own coordinate system– Different parts of the VW can be designed
independently
Nicolas D. Georganas
SPLINE: Locales• Each locale has a list of adjacent locales
– This list includes coordinate transformations as well
– Portals (non-Euclidean motion)• Locales may overlap, be enclosed or contain
empty spaces
Nicolas D. Georganas
Presentation
• Virtual Reality & Collaborative Virtual Environments
• Applications• Standards• Performance Issues
Nicolas D. Georganas
SPLINE - Performance
44.4688.8
170.65239.23
300.3
0
50
100
150
200
250
300
350
1 25 50 75 100
spWMUpdate Refresh Interval
End-
to-E
nd D
elay
Ownership transfer between 2 clients connected in confined 10Base2 Network (VRML browser access via JNI and DLL)
Nicolas D. Georganas
Performance for Various CVE Middleware Standards
RTI with VRML
RTI with Java3D
JSDT SPLINE with
VRMLCCD (Client-to-Client Delay)
132 112 - 129
CommunicationDelay *
89 89 60 85
OTD (Ownership Transfer Delay)
225 192 5 217
* This is CCD minus the delays between the middleware and the rendering part (VRML/Java3D).
Nicolas D. Georganas
The Updates Distribution Problem• When the number of participants in a CVE grows
sufficiently, the amount of updates each participant has to deal with becomes overwhelming.
• Some architectures proposed allow a large number of users to collaborate; however they all fail under certain conditions.
• It is usually very hard to balance the load in the various systems participating in a VE simulation, yet worst if a great range of systems with different processing power is employed (heterogeneous systems):
– Everything is usually handled based on the slowest system; or
– Any system which does not provides a minimum capacity is simply denied access to the session
Nicolas D. Georganas
Limitation of Existing Models
• Space based solutions assume that users are uniformly spread in the CVE.
– Reducing the area you see must reduce what you see...
Nicolas D. Georganas
Limitation of Existing Models
• It fails to filter incoming dataif a sufficiently large amountof users is packed togetherin a single “locale”.
• Space based solutions assumethat users are uniformly spreadin the CVE.
– Reducing the area you see must reduce what you see… Not necessarily...
Nicolas D. Georganas
Limitation of Existing Models
• All users in the same area receive the same load, which overloads and/or under uses heterogeneous systems.
Nicolas D. Georganas
VELVET: An Architecture for Very Large Virtual Environments
(DISCOVER Lab, UofO)
• Adaptive Architecture– Allows the system to adapt itself according to the load.– Allows diverse CPU powered stations to participate in
the simulation.– Best effort approach (a user will “see” as much as
his/her station allows)
• Hybrid Architecture– Builds on top of space based (i.e., at light load it may
work just like a space-based solution, such as SPLINE)
Nicolas D. Georganas
VELVET - Best Effort Adaptation
Nicolas D. Georganas
VELVET - Best Effort Adaptation
Nicolas D. Georganas
VELVET - Best Effort Adaptation
Nicolas D. Georganas
VELVET - Best Effort Adaptation
Nicolas D. Georganas
VELVET - Best Effort Adaptation
Nicolas D. Georganas
Check-in/Check-out• When an object crosses the edge of the Area of Interest
it checks-in/out.• There is unnecessary processing performed if a given
object stays at the border for a long time with light motion back and forth, which generates of large amount of check-in/out operations
• VELVET defines two Areas of Interest aiming at avoiding the problem described above
– AI-Check-In (AICI)– AI-Check-Out (AICO)
Nicolas D. Georganas
Check-in/Check-out
AICOAICI
Nicolas D. Georganas
Check-in/Check-out
AICOAICI
Nicolas D. Georganas
Check-in/Check-out
AICOAICI
Nicolas D. Georganas
Check-in/Check-out
AICOAICI
Nicolas D. Georganas
Check-in/Check-out
AICOAICI
Nicolas D. Georganas
Check-in/Check-out
AICOAICI
Nicolas D. Georganas
Check-in/Check-out
• The distance between AICI and AICO may be controlled in such a way to increase if a given number of Check-In/Out operations are performed aiming at reducing them in the future and vice-versa.
Nicolas D. Georganas
Parallel Virtual World• A way to see the world
based on what matters the most, rather than what is geometrically closer (Euclidean space).
– World is organized according to a pre-defined metric.
– Area of Interest is actually defined in the PVW, rather than the CVE itself.
Nicolas D. Georganas
Simulation Results
Nicolas D. Georganas
Conclusion• Distributed and Collaborative Virtual
Environments open a new avenue for multimedia human interaction, collaboration and learning
• Many Applications:– Defense Training– Medical training – Industrial Training, – Telecollaboration, – E-commerce,….
• Phychological and social issues: an open area for study.
Nicolas D. Georganasمتشکرم