Date post: | 02-Jan-2016 |
Category: |
Documents |
Upload: | sarah-cooper |
View: | 27 times |
Download: | 4 times |
Miroslav Andel - Development of immersive real-time applications
2
• Development of immersive real-time applications– Online/direct rendering– Hardware– Programming
• Video production– Offline rendering– Hybrid solutions
Lectures
2014-09-03
Miroslav Andel - Development of immersive real-time applications
3
• Hardware– Understanding the technology– Clusters and GPUs– Warping, color matching & blending
• Immersion in a virtual environment (VE)• Software design
– Desktop application vs. immersive cluster application– Scalability & configuration of VEs– Frameworks– Scene graph
Outline
2014-09-03
Miroslav Andel - Development of immersive real-time applications
4
Hardware
2014-09-03
Miroslav Andel - Development of immersive real-time applications
5
• 6x Digital Projection Titan• 1920 x 1200 @ 120 Hz• Frame doubling
Projectors
2014-09-03
Miroslav Andel - Development of immersive real-time applications
6
Image Generators (IGs)
2014-09-03
– Workstation high-end graphics cards• Frame sync• External house sync• Genlock (video signal sync)• Swaplock (swap buffer sync)• 3D sync (for 3D glasses)
– Supports active stereoscopy using OpenGL– Nvidia Quadro or AMD/ATI FirePro– No gaming cards
Miroslav Andel - Development of immersive real-time applications
7
• GPU is pretty much the same but different driver software and firmware
• Gaming cards are optimized for speed and DirectX– Hi frame-rate in games, highly optimized
• Workstation cards are optimized for precision, OpenGL & OpenCL– Designed for 3D modeling, CAD, video editing and computation– Higher floating point and sub-pixel precision– Better quality and larger memory (12 GB in Quadro K6000, 16 GB AMD
FirePro 9100)• Ray tracing on the GPU
• Our problem: We use gaming techniques but need workstation features like synchronization and stereoscopy (less performance)
Gaming vs. Workstation GPU
2014-09-03
Cluster of IGs
• Often consists of workstations with the same specification
• Master – Slave• In our case the master has an
additional surround sound card (live encoding to Dolby Digital or DTS)
Miroslav Andel - Development of immersive real-time applications
8
Clusters(at the visualization center)
• Dome– Alpha (primary real-time cluster used for shows)– Beta (secondary development cluster)– Delta (movie and presentations playback cluster)– Gamma (single node eyefinity machine)
• VR-Arena– Theta– Videowall (just a single node with 6 outputs)
used for movie and presentations playback
Miroslav Andel - Development of immersive real-time applications
9
Miroslav Andel - Development of immersive real-time applications
11
• What is white?
Color/gammut matching
2014-09-03
Miroslav Andel - Development of immersive real-time applications
15
• Digital blending – good bright levels– Not a linear gradient (gamma)– Done after color and brightness matching
Blending
2014-09-03
Miroslav Andel - Development of immersive real-time applications
16
• Optical blending for better dark levels– Projected black is not black, still projected light
Blending
2014-09-03
Miroslav Andel - Development of immersive real-time applications
17
• Optical blending for better dark levels– Projected black is not black, still projected light
Blending
2014-09-03
Miroslav Andel - Development of immersive real-time applications
18
• Blending in the dome– Blue = Projection borders– Red = Blend zones
Blending
2014-09-03
Miroslav Andel - Development of immersive real-time applications
19
• Correct the rendered frame to match the geometry of the screen, lens and projector’s position
• Correct overlaps between adjacent channels• Spatial 2D transform
Warping/geometric correction
2014-09-03
Miroslav Andel - Development of immersive real-time applications
20
• Processing unit with input & outputs cards (FPGA) with minimal latency
Warping and blending hardware
2014-09-03
Miroslav Andel - Development of immersive real-time applications
21
• No latency but uses a bit of the GPU
Warping and blending using OpenGL
2014-09-03
Miroslav Andel - Development of immersive real-time applications
22
• Manual warping and blending calibration takes many hours– Few reference points in the dome– Moving mesh points by hand causes linearity problems
• Semi-automated calibration• Automated camera-based systems
– Current systems don’t handle optical blends
Calibration process
2014-09-03
Miroslav Andel - Development of immersive real-time applications
23
Hardware (end)
2014-09-03
Miroslav Andel - Development of immersive real-time applications
24
• To give the audience the impression that they have stepped into a synthetic world
Immersion
2014-09-03
Miroslav Andel - Development of immersive real-time applications
25
• To give the audience the impression that they have stepped into a synthetic world
• Enhance immersion by:– Large field of view– High resolution– Stereoscopy (can also reduce immersion)– Surround sound– Smooth navigation (based on a physical model)– Realistic rendering (shadows etc., to improve depth cue)
Immersion
2014-09-03
Miroslav Andel - Development of immersive real-time applications
26
• Stereoscopy in the dome?– The paradox– Field of view wearing 3D glasses– Use stereoscopy wisely
Immersion
2014-09-03
Miroslav Andel - Development of immersive real-time applications
28
Software design
2014-09-03
Miroslav Andel - Development of immersive real-time applications
29
• Several projectors/screens– A lot of pixels– Alignment, blending & warping– Color & Brightness matching
• Several computers (cluster)• Synchronization
– Frame– Buffers swap– Data– Video signal (genlock)
• Multi-threading
The difference(from normal desktop application development)
2014-09-03
Miroslav Andel - Development of immersive real-time applications
30
• How does it work?
Stereoscopy
2014-09-03
Miroslav Andel - Development of immersive real-time applications
31
Stereoscopy
2014-09-03
Miroslav Andel - Development of immersive real-time applications
32
Stereoscopy
2014-09-03
Miroslav Andel - Development of immersive real-time applications
33
Stereoscopy
2014-09-03
Miroslav Andel - Development of immersive real-time applications
34
• Co-location– Camera position = eye position– Projection plane = real physical screen
Stereoscopy
2014-09-03
Miroslav Andel - Development of immersive real-time applications
35
• Don’t change eye separation, change the scale of your scene
• ~15 meters limit
Stereoscopy
2014-09-03
Miroslav Andel - Development of immersive real-time applications
36
2014-09-03
Miroslav Andel - Development of immersive real-time applications
37
2014-09-03
Miroslav Andel - Development of immersive real-time applications
38
2014-09-03
Miroslav Andel - Development of immersive real-time applications
39
• Object in focus in negative parallax
Stereoscopy
2014-09-03
Miroslav Andel - Development of immersive real-time applications
40
Stereoscopy
2014-09-03
• Object in focus in positive parallax
Miroslav Andel - Development of immersive real-time applications
41
• Problems with a fixed head setup (no head tracking)
Stereoscopy
2014-09-03
Miroslav Andel - Development of immersive real-time applications
42
• One application can run in several virtual environments without the need of changing the code– In the dome– In the VR-Arena– In the VR-Lab– At home using your computer– At the bus using a laptop
• If the platform is the same then there is no need for recompiling the application (just copy & paste)
Scalability
2014-09-03
Miroslav Andel - Development of immersive real-time applications
43
The application uses a configuration file that describes the hardware in terms like:• Number of IGs• Number of displays• Number of viewports• Physical location of displays/projection planes• Physical location of the user• Eye separation of the user• Type if Stereoscopy• Etc…
Scalability
2014-09-03
44
• “3d drawing”
Configuration
2014-09-03
45
Configuration
2014-09-03
Immersive applicationsOften uses non-symmetric frustums/projections, while normal OpenGL applications don’t (left FOV = right FOV, top FOV = bottom FOV)
Monoscopic projection
46
Configuration
2014-09-03
Stereoscopic projection
Configuration47
Configuration48
All users except the target user will perceive the rendered object incorrectly.
Configuration49
All users except the target user will perceive the rendered object incorrectly.
One solution would be using HMDs or a multi-view screen.
Miroslav Andel - Development of immersive real-time applications
50
• In a desktop application a user navigates by moving the camera
Navigation
2014-09-03
Miroslav Andel - Development of immersive real-time applications
51
• In a desktop application a user navigates by moving the camera
• In a scalable application the number of cameras/frustums is not known
Navigation
2014-09-03
Miroslav Andel - Development of immersive real-time applications
52
• In a desktop application a user navigates by moving the camera
• In a scalable application the number of cameras/frustums is not known
• Instead of moving the user and all “cameras” in the same direction the whole scene is moved in the opposite direction
Navigation
2014-09-03
Miroslav Andel - Development of immersive real-time applications
53
• Will help you with– Cluster setup & synchronization– Frustums & viewports– Input– Tracking
Frameworks(the good stuff)
2014-09-03
Miroslav Andel - Development of immersive real-time applications
54
Frameworks
2014-09-03
• VR Juggler
• CAVElib (commercial)• Equalizer
Miroslav Andel - Development of immersive real-time applications
55
• Massive and have a lot of dependencies• Big initial effort to get started• Often in legacy OpenGL• Takes usually several days to get started
– Getting all dependencies– Compile all dependencies– Compile the whole framework– Read the documentation
Frameworks(the bad stuff)
2014-09-03
Miroslav Andel - Development of immersive real-time applications
56
SGCTSimple Graphics Cluster Toolkit
2014-09-03
Miroslav Andel - Development of immersive real-time applications
57
• Static library based on– GLFW– GLEW– GLM– Freetype– TinyThread++– TinyXML– PNG– Z– VRPN– TurboJPEG
SGCTSimple Graphics Cluster Toolkit
2014-09-03
Miroslav Andel - Development of immersive real-time applications
58
• Developed here and is free for everybody to use• Designed to get started immediately
(rapid development)• Cross-platform C++• Modern and Legacy OpenGL support (OGL 2.2 – 4+)• Single static library (no dll-hell)• Using callbacks rather than
inheritance/polymorphism– Easy porting application from GLUT, GLFW or similar
SGCTSimple Graphics Cluster Toolkit
2014-09-03
Miroslav Andel - Development of immersive real-time applications
59
• Wiki at: c-student.itn.liu.se– Documentation– Tutorials– Configuration files– Downloads (installers)– Support
• Forum (on it’s way)
SGCTSimple Graphics Cluster Toolkit
2014-09-03
Miroslav Andel - Development of immersive real-time applications
60
• Has an engine which handles– Initiation of OpenGL– Rendering– Synchronization– Network communication– Configuration handling– Input
• The user can bind callbacks (function pointers) to the engine to customize the functionality
SGCT – How it works
2014-09-03
61
2014-09-03
Miroslav Andel - Development of immersive real-time applications
62
• Set only the callbacks you need• A minimal cluster application contains:
– Draw callback: Render your stuff– Pre Sync callback: generate the data to be shared– Encode callback: Send the data to the slave(s)– Decode callback: Receive the data from the master
• Other callbacks to handle input devices and external GUIs
SGCT – How it works
2014-09-03
63
• Frame-, swapbuffer- and application data-synchronization
• 3D/Stereoscopic rendering– Active/Quadbuffer stereo– Side-by-side stereo– Top-bottom stereo– DLP/Checkerboard stereo– Line interleaved stereo– Anaglyph stereo
SGCT - Features
2014-09-03 Miroslav Andel - Development of immersive real-time applications
Miroslav Andel - Development of immersive real-time applications
64
• Fisheye rendering• PNG texture management• GLSL shader management• Freetype font management• Error messaging across the cluster• External TCP control interface• Tracking support• Joystick, gamepad, Kinect, steering wheel etc. input• Antialiasing• 3D screenshots in maximum texture resolution• Overlays• Statistics and performance graphs
SGCT - Features
2014-09-03
Miroslav Andel - Development of immersive real-time applications
65
• XML configuration• All nodes are using the same configuration and
can automatically determine if they are a slave or a master
SGCT - Configuration
2014-09-03
66<?xml version="1.0" ?><Cluster masterAddress="127.0.0.1">
<Node ip="127.0.0.1" port="20401"><Window fullscreen="false">
<Size x="640" y="360" /><Viewport>
<Pos x="0.0" y="0.0" /><Size x="1.0" y="1.0" /><Viewplane>
<!-- Lower left --><Pos x="-1.778" y="-1.0" z="0.0" /><!-- Upper left --><Pos x="-1.778" y="1.0" z="0.0" /><!-- Upper right --><Pos x="1.778" y="1.0" z="0.0" />
</Viewplane></Viewport>
</Window></Node><User eyeSeparation="0.065">
<Pos x="0.0" y="0.0" z="4.0" /></User>
</Cluster>
Miroslav Andel - Development of immersive real-time applications
67
• Normal use: app.exe –config “config.xml”
• Running cluster application locally as master:app.exe –config “cluster_config.xml” –local 0
• Running cluster application locally as slave:app.exe –config “cluster_config.xml” –local 0 --slave
SGCT arguments
2014-09-03
Miroslav Andel - Development of immersive real-time applications
68
• Command prompt/terminal• Create a Batch file (.bat)• Create a Shortcut• From IDE
Running SGCT
2014-09-03
Miroslav Andel - Development of immersive real-time applications
69
• The application folder needs to be accessible from all IGs/nodes1. Copy the whole folder
• Use Microsoft’s robocopy tool for mirroring folders across a cluster
2. Use a network share
• DemoManager tool for launching (and killing) processes across a cluster
• Or manually start the application on all nodes
Running cluster applications
2014-09-03
Miroslav Andel - Development of immersive real-time applications
70
SGCT – Example
2014-09-03
• Basic spinning triangle synchronized across a cluster using less then 80 lines of code (including comments and spaces)
Miroslav Andel - Development of immersive real-time applications
71
• What to share?– Time or dt– Run simulations only on master or on whole
cluster?– Random numbers?– Minimize traffic– Compression– Bit-masking
Data synchronization
2014-09-03
Miroslav Andel - Development of immersive real-time applications
72
• OpenGL Mathematics (GLM) is included in SGCT• Computer graphics is linear algebra!• GLM makes life easy
– Vectors– Matrices– Quaternions
• Similar syntax to glsl
Math
2014-09-03
Miroslav Andel - Development of immersive real-time applications
73
• Local cluster on a single computer• View
Testing anywhere
2014-09-03
Miroslav Andel - Development of immersive real-time applications
74
Testing anywhere
2014-09-03
Miroslav Andel - Development of immersive real-time applications
75
Open Scene Graph (OSG)• www.openscenegraph.org• Feature rich• SGCT manages
– Cluster sharing & synchronization– Window, frustum and viewport creation
• OSG manages– Content
• OSGMaxExporter (3Ds Max plugin)
SGCT + OSG
2014-09-03
Miroslav Andel - Development of immersive real-time applications
76
Scene graph
2014-09-03
• Tree structure of nodes• Intelligent structure
– Visibility & occlusion culling
Miroslav Andel - Development of immersive real-time applications
77
OSG example
2014-09-03
Miroslav Andel - Development of immersive real-time applications
78
• Positional 3D audio– OpenAL– Fmod Ex
Audio
2014-09-03
Miroslav Andel - Development of immersive real-time applications
79
• Video pipeline• Offline rendering for the dome
Next time
2014-09-03