Visual Communication in the AQUA environment

transcript

Technical aspects and solutions

Stefan Seipel Lars W. Pettersson

Björn Andersson

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

We are situated in a

co-located multiple viewing environment

Co-located multiple viewing environments Example: iRoom [Fox2000]

Based on conventional 2D GUI

http://graphics.stanford.edu/projects/iwork/room/images/using-room-feb-00/

Co-located multiple viewing environments The AQUARIUM [Sundin2000]

Beyond the 2D graphical user interface

Why is 3D so interesting ?

Example of what could be done with 3D representations

Do existing tools fix the job?

NetVR Full transparency and SG/DB replication

Latency issues not a predominant issue

Concurrency of media streams more important

Cluster/Tiled Rendering Optimization

Single data multiple processes

Single data multiple displays

AQUARIUM – A 3D Interactive Environment

Specific requirements

Tiled displays

Multiple view ports

Image superimposition

Multiple views on a shared 3D scene

Individual views on private data

Co-location of multiple displays

Tiled displays

Multiple view ports

Tiled displays

Multiple view ports

Tiled displays

Multiple view ports

Tiled displays

Multiple view ports

How to design applications…

…that support “20 eyes upon 15 visuals” ?

Multiple processes

Allow for flexible configuration

Management of visual views

Provide multiple interaction contexts

Easy to use programming concepts

Controlling by sharing data

Shared State Repository [Lindkvist2001]

Client 1

Client 2

Client n

Server

Virtual Shared Memory

TCP/UDP

• Client-Server architecture

• Lean and easy to use API

• Based on TCP, UDP or MC

• Small data subscription packages

• No concurrence control

srAlloc

srFree

srUpdate

Data Pools Pooling context relevant data

Application

VR-Tool

Low Level 3D (OpenGL)

Rendering HW

Display

AQUARIUM application

Application

VR-Tool

Low Level 3D (OpenGL)

Rendering HW

Display

AQUARIUM application

projector pool

pipe pool

node pool

Data Pools Pooling context relevant data

projector pool

pipe pool

node pool

Viewing frustumHead position

Real world metrics

where?Buffer specificPixel metrics

Reference to a projector

An Example Scenario

Retroscope1

Retroscope2

Vsionarium1

Vsionarium2

Tracker 1

ViewManager

What about performance?

Latency and frame incoherence will introduce

visual artifacts!

Little research done

Edge discontinuity

Hyper- or hypo-parallax

Vertical parallax

Peripheral viewing artifacts

General test set-up:

Client 2

Render

Client 1

Render

Server 1

Capture&Analysis

2 Rendering processes

1 Animation process

Simple scene

Register output

Analyze differences

Observation: # frames out-of-sync.

Test A : Local host

server

200 framesAVI Sequence

Client2800x300

Client1800x300

framegrabber800x600

Snurr.wmv

1 animated nodeva = 90°/sec. const.Animation rate: 10Hz, 20Hz, 30Hz, 40Hz, 50Hz

stefan:stefan:

fr~80Hz

Host 1

DifferentialAnalysis

PIII/2.2GHz, GeForce4 4600

Test B : Local area network

Client2800x300

Client1800x300

framegrabber800x600

Snurr.wmv

1 animated nodeva = 90°/sec. const.Animation rate: 10Hz, 20Hz, 30Hz, 40Hz, 50Hz

stefan:stefan:

fr~80Hz

server

Host 1Host 2

LAN10/100

MBit DifferentialAnalysis

PIII/2.2GHz, GeForce4 4600PIII/500MHz, E&S

Results for Test A and Test B

stefan:stefan:

l host

Animation rate

Frame number

#differing pixels

Results for Test A and Test B

stefan:stefan:

Animation Rate 10Hz 20Hz 30Hz 40Hz 50HzAsync. Frames A (%) 0,5 3,5 3 5,5 4,5Async. Frames B (%) 1,5 1,5 1 2,5 5Async. Frames A 1 7 6 11 9Async. Frames B 3 3 2 5 10

10Hz 20Hz 30Hz 40Hz 50Hz

Async. Frames A (%)

Async. Frames B (%)

Test C and D : Traffic

va = 90°/sec. const.

stefan:stefan:

Client2800x300

Client1800x300

framegrabber800x600

Snurr.wmv

fr~80Hz

server

Host 1Host 2

LAN10/100

MBit DifferentialAnalysis

LAN configuration as in Test B

Increasing the number of shared states

Test C: (10 Hz; 1,10,20,40,60,80,100,120,140,160,180,200)

Test D: (20 Hz; 1,10,20,40,60,80,100,120,140,160,180,200)

PIII/2.2GHz, GeForce4 4600PIII/500MHz, E&S

Results for Test C and Test D

stefan:stefan:

Number of nodes 1 10 20 40 60 80 100 120 140 160 180 200Async. Frames C (%) 2 2 0,5 0,5 0,5 2 2 3,5 2,5 2,5 3,5 3Async. Frames D (%) 0,5 2 2 2,5 2,5 1,5 1,5 1,5 1 1 2 2Async. Frames C 4 4 1 1 1 4 4 7 5 5 7 6Async. Frames D 1 4 4 5 5 3 3 3 2 2 4 4

1 10 20 40 60 80 100 120 140 160 180 200

Async. Frames C (%)

Async. Frames D (%)

Conclusion

Framework allows for building modular complex visualization environments

Flexible to use and combine with existing tools

No need for fancy protocols to maintain adequate frame-sync.

The fusion 2D and 3D Interfaces

The AQUARIUM is a 3D environment

How can legacy code be re-used?

Can 2D applications be instantiated

within a 3D environment?

Virtual Network Computing

System for sharing frame-buffers/applications

Developed by AT&T Laboratories Cambridge,

Open Source

Remote Frame Buffer Protocol well documented

Servers and clients readily available for Microsoft

Windows, Unix, Linux och MacOs

Bold servers, thin clients

2D Virtual Network Computing Client

A 3D VNC Client

VNC Server

3D VNC Client 3D VNC Client 3D VNC Client

• Decode RFB protocol

• Administrate local texture buffers

• Handle 3D input and map to 2D

server coordinates

x’ y’

3D VNC Application

Example in the AQUARIUM

Contextual browsing of auxiliary information on the WEB

3D VNC Performance Assessment Benchmarking

Frame rate Delay

Common Interaction Text editing Mouse movement Web browsing Streaming videos

Modular Application Development

Background The framework & component design Communication between components

The VASE development framework

Background Until now – monolithic application Different but similar applications

Many components reappear in several of the applications

Makes modular development natural Framework that handles this is needed

(VASE) Easy script (XML)

The modular approach enables a parallel development process

Middlewares used VRT – Virtual Reality Toolkit

For graphics handling Built on top of OpenGL Implements a scenegraph

STREEP For network communication Shared state repository Supports both TCP and UDP

Design of the framework and the basic structure of the plugins

The framework defines the basic structure for the components (plugins)

Implements a main module Each plugin is written as a DLL and

implements a class, which structure is inherited from a base class

Plugins are dynamically linked during execution

Easy to change or create new plugins when new functionality is to be added

Main module

Plugin DLL

Class implementation

Plugin instance

The main module The main module (vase.exe) handles:

Parsing of a configuration file (*.vas) Loading and creation of plugin instances Distributes user interaction events Handles message passing Handles shared states

”myfile.vas” <component type=vpiAvatar name=”client>

</component>Huvudmodul (VASE.exe)

Plugin1 Plugin2 Plugin3

Plugin pool

Plugin DLLs

Example configuration file

<projector>TABLET_PC_VIEW <width>207</width> <height>138</height> <front>50</front> <back>100</back> <window_center>0.0 0.0 0.0</window_center> <window_normal>0.0 1.0 0.0</window_normal> <window_up>0 0.0 -1.0</window_up> <view_point>0.0 200.0 0.0</view_point></projector>

<pipe>TABLET_PC_PIPE <projector>TABLET_PC_VIEW</projector> <left>0</left> <bottom>0</bottom> <width>1024</width> <height>768</height></pipe>

</component>

Client 2Client 1

Communication Local communication

Function calls to the main module

Method calls

Network communication Uses STREEP Through the main module Message passing

Receiving plugin concatinated to the message

Shared states Subscription lists keeps track

of who is interested in what

Main module

Plugin1 Plugin2

Plugin1

Shared state repository

Main moduleMain module

Plugin2

Example of Aqua components (plugins) AquaController

3D-buttons Sends messages to other plugins

AquaEchelon Draws an echelon graph Listens to parsed Stratmas data Implements an internal XML structure for units

AquaEnvironment Handles the drawing of the

map grid

AquaParser Inuput: Stratmas generated data file Output: Parsed binary file Distributes the parsed data to other plugins

AquaViewer Handles projectors and pipes Tracking devices

AquaWhiteboard Overlay for the map Stores strokes in RT90 coordinates

How do we connect the AQUA environment to the rest of the world?

The STRATMAS-Link Modifications to STRATMAS Socket connectivity File connectivity XML encoding

The STRATMAS Link- Modifications to STRATMAS

TCP socket communication add in a thread

conf.dat (link to) ID number for each Unit

The STRATMAS Link- Socket connectivity

Connection on an hostname and port specified in conf.dat

XML data is sent in fixed sized blocks over the socket connection

Communication is one way

The STRATMAS Link- File connectivity

Generation of data.txt on the Mac running STRATMAS Xml encoded (see next slide)

The STRATMAS Link- XML for Units

int ID;long superior_ID; // -1 if no superiorint rank; // 0-6 - enum RankTypefloat latitude;float longitude;int purpose; // 0-13 - enum PurposeTypeint vehicle_type; // 0-6 - enum VehType, -1 if no vehicleint unit_type; // 0-8 - enum UnitTypeint condition; // 0-100int health; // 0-2 - enum HealthTypelong my_threat; // Larger value - larger threatfloat sens_range; // Sensor range in km, -1 if no sensorsfloat veh_sens_range; // Vehicle sensor range in km, -1 if no vehiclefloat weapon_range; // Shoot radius in km, -1 if no vehiclelong shots_fired; // Number of shots firedlong fi_injured; // Number of enemies injuredlong fi_killed; // Number of enemies killedfloat vx; // vehicle velocity vector in degrees *lng* per

// iteration, -1 if no vehiclefloat vy; // vehicle velocity vector in degrees *lat* per // iteration, -1 if no vehiclefloat speed_kph; // vehicle speed in kilometers per hour, -1 if no vehiclefloat goal_x; // goal position RT90xfloat goal_y; // goal position RT90y

int flag; // Flag for red, green or blue unit. Values: 0, 2 or 1.

The STRATMAS Link- XML for Gridint rows; // Number of rows in the grid.int cols; // Number of columns in the grid.int nc; // Number of cells holding relevant data.int nlayers; // Number of layers. char lname[MAX_GRID_LAYERS][64]; // Layer names. int *index; // Array of the cell-indices corresponding to the data

array indices float *data; // The values for each layer are stored sequencially

after eachother. for red, green or blue unit. Values: 0, 2 or 1.

Map View

Topographic context Based on RT90 coordinate reference Mapped upon internal grid

1200000

1500000

1900000

6100000

7700000

15º48’29.8’’Ö

50kmx50kmScale 1:100.000

28 x 64 cells á 25x25 km (1:25000)

700000/0

700000/16000000/1600000

Map View contd. 2 textures

1024x1024texels

Map cell25 x 25 km32 x 32 texels1 texel ~ 780 meters

700000 [meters (RT90)]

Modeling unit in the virtual environment is 1cm (1:1000000)-> Map (70cmx160cm) fits visionarium

Map View summary

Map granularity can be adapdedAt present topography is schematic

High resolution map data can be loaded dynamicallyper map cell (e.g. 1:25000 maps per cell)depending on zoom level

Advantage of RT90 reference frameinternal representation of 3D data always Euklideanorthogonal system eases texture mappinguse lat/long conversion routines by metria

Cell Grid Layer Co-located with the RT90 map

Mapps geographically related information upon map

Grid cell contain aggregated data for military units

Grid cell contain substrate information

Grid cell can visualize itself

Grid size and resolution can be chosen arbitrarily

Current limits:

Max cell size: 200x200 km2

=> 32 cells/RT90 map

Min cell size: 3.125 x 3.125 km2

=> 114688 cells/RT90 map

Cell Properties

Cell geometry (polygon)

Cell texture

Cell related data

num_red : 0, 1,2,3,4,5 => number of red troups in 5 levels 0 = no red troups

num_blue: 0, 1,2,3,4,5 => number of blue troups in 5 levels 0 = no red troups

num_green: 0, 1,2,3,4,5 => number of green troups in 5 levels 0 = no green troups

age: 0, 1,2 => 0 = very up-to-date information; 1 = older; 2=very old

sensor: 0, 1 => 0 = there is no sensor coverage; 1 = sensor coverage

disease: 0, 1,2 => 0 = no disease; 1 = disease1; 2 = disease2

strenght_red: 0, 1,2,3 => 0 = perfect strenght, 1,2,3 dicreasing moral/strenght

strenght_blue: 0, 1,2,3 => 0 = perfect strenght, 1,2,3 dicreasing moral/strenght

strenght_green:0, 1,2,3 => 0 = perfect strenght, 1,2,3 dicreasing moral/strenght

combat: 0, 1,2 => 0 = no fight; 1 = little fight; 2 = heavy fight

Mapping Data from external Sources (e.g. Stratmas)

Continuous data(unit data)

Discrete data(raster data)

. . ... .

Stratmas gridrows="79”cols="36"

CellGrid::SetStratmasGridData

CellGrid::SetStratmasUnitData

Visualizing Cell Data

Visual pattern library

Patterns associated with properties (32x32 texels)

Patterns can be combined

Examples:

Cell property Visual pattern(s)

Number of troups (ordinal)

No sensor coverage

Diseases

Troup strength

Combat

Visualizing Cell Data contd. Rules, based on actual cell data

- Chosing the correct base-pattern- Applying modifiers- Texture combiner

Examples of precompiled patterns (512x512 texels)

Visualizing Cell Data contd.

3 forces (red, green, and blue)

3 disease conditions

3 x 3 x 8 megabye/texture = 75 MB texture memory

This is equivalent to 2304 unique patterns

One more attribute will double the memory requirements

This is quite unlikely there are only 1800 grid cells in the default resolutionnot all cells contain interesting data

Cell pattern compilation at run-time for individual cells

Cell Grid Visualization Example

Visualization of continuously positioned data – Echelon graphs

AquaEchelon in detail Function

Draws tactical information in 3D Gives both a spatial and hierarchical

understanding at a glance Interactable

XML Bottom-up vs. Top-down

The XML formats Internal format

Top-down Shows a static situation

Example of the internal XML format<echelon> <unit type="skytte" size="pluton" pos="1550000 6600000 "> <weapon type="grk" force="5" range="7"/> <sensor type="radar" range="20"/> </unit>

The XML formats – cont. Stratmas generated

Bottom-up Dynamic, can play whole scenarios

Example of a Stratmas generated file<stratmas> <units flag="red" count=“2"> <unit delay="0"> 1 -1 6 59.383562 17.027027 0 0 0 0.00 0 0.0 0.0 0.0 0.0 0 0 0 0.0 0.0 0.0

59.383562 17.027027 0 </unit> <unit delay="0"> 2 1 5 59.383562 17.027027 0 0 0 0.00 0 0.0 0.0 0.0 0.0 0 0 0 0.0 0.0 0.0

59.383562 17.027027 0 </unit> </units></stratmas>

Symbol construction Previously manually generated bitmaps

Now 3D-models of the basic shapes

Symbol construction

. . ..

Construction of the symbols Make 3D-models of the basic shapes Merge several of these into a complete unit

Easy to alter the apperance of the symbols by changing the basic models

3D instead of 2D

Overlays

Overlays Storage data structure

A document... • Has one or several pages...

• A page has none or several strokes

Rendering in a bitmap texture Strokes are rendered in their correct RT90 context

File format Text/Ascii based format.

Interaction in with the AQUA environment

Demo...

Visual Communication in the AQUA environment

Documents