Visual Communication in the AQUA environment

Visual Communication in the AQUA environment

Technical aspects and solutions

Stefan Seipel Lars W. Pettersson

Björn Andersson

Uppsala University

Info

rmati

onst

ekn

olo

gi

Institutionen för informationsteknologi | www.it.uu.se

We are situated in a

co-located multiple viewing environment

Info

rmati

onst

ekn

olo

gi


Co-located multiple viewing environments Example: iRoom [Fox2000]

Based on conventional 2D GUI

http://graphics.stanford.edu/projects/iwork/room/images/using-room-feb-00/

Info

rmati

onst

ekn

olo

gi


Co-located multiple viewing environments The AQUARIUM [Sundin2000]

Beyond the 2D graphical user interface

Info

rmati

onst

ekn

olo

gi


Why is 3D so interesting ?

Example of what could be done with 3D representations

Info

rmati

onst

ekn

olo

gi


Do existing tools fix the job?

NetVR Full transparency and SG/DB replication

Latency issues not a predominant issue

Concurrency of media streams more important

Cluster/Tiled Rendering Optimization

Single data multiple processes

Single data multiple displays

Info

rmati

onst

ekn

olo

gi


AQUARIUM – A 3D Interactive Environment

Specific requirements

Tiled displays

Multiple view ports

Image superimposition

Multiple views on a shared 3D scene

Individual views on private data

Co-location of multiple displays

Info

rmati

onst

ekn

olo

gi




Tiled displays

Multiple view ports





Info

rmati

onst

ekn

olo

gi




Tiled displays

Multiple view ports





Info

rmati

onst

ekn

olo

gi




Tiled displays

Multiple view ports





Info

rmati

onst

ekn

olo

gi




Tiled displays

Multiple view ports





Info

rmati

onst

ekn

olo

gi


How to design applications…

…that support “20 eyes upon 15 visuals” ?

Multiple processes

Allow for flexible configuration

Management of visual views

Provide multiple interaction contexts

Easy to use programming concepts

Info

rmati

onst

ekn

olo

gi


Controlling by sharing data

Shared State Repository [Lindkvist2001]

Client 1

srAPI

Client 2

srAPI

Client n

srAPI

Server

srAPI

Virtual Shared Memory

TCP/UDP

• Client-Server architecture

• Lean and easy to use API

• Based on TCP, UDP or MC

• Small data subscription packages

• No concurrence control

srAlloc

srFree

srGet

srUpdate

...

Info

rmati

onst

ekn

olo

gi


Data Pools Pooling context relevant data

Application

VR-Tool

Low Level 3D (OpenGL)

Rendering HW

Display

AQUARIUM application

srAPI

Application

VR-Tool

Low Level 3D (OpenGL)

Rendering HW

Display

AQUARIUM application

srAPI

projector pool

pipe pool

node pool

Info

rmati

onst

ekn

olo

gi


Data Pools Pooling context relevant data

projector pool

pipe pool

node pool

how?

Viewing frustumHead position

Real world metrics

where?Buffer specificPixel metrics

Reference to a projector

what?

Info

rmati

onst

ekn

olo

gi


An Example Scenario

Retroscope1

Retroscope2

Vsionarium1

Vsionarium2

Tracker 1

ViewManager

Info

rmati

onst

ekn

olo

gi


What about performance?

Latency and frame incoherence will introduce

visual artifacts!

Little research done

Edge discontinuity

Hyper- or hypo-parallax

Vertical parallax

Peripheral viewing artifacts

Info

rmati

onst

ekn

olo

gi


General test set-up:

pool

Client 2

Render

Client 1

Render

Server 1

Capture&Analysis

2 Rendering processes

1 Animation process

Simple scene

Register output

Analyze differences

Observation: # frames out-of-sync.

Info

rmati

onst

ekn

olo

gi


Test A : Local host

server

20 Hz

200 framesAVI Sequence

fan

Client2800x300

Client1800x300

framegrabber800x600

Snurr.wmv

1 animated nodeva = 90°/sec. const.Animation rate: 10Hz, 20Hz, 30Hz, 40Hz, 50Hz

stefan:stefan:

fr~80Hz

fr~80Hz

Host 1

DifferentialAnalysis

PIII/2.2GHz, GeForce4 4600

Info

rmati

onst

ekn

olo

gi


Test B : Local area network

20 Hz


Client2800x300

Client1800x300

framegrabber800x600

Snurr.wmv

1 animated nodeva = 90°/sec. const.Animation rate: 10Hz, 20Hz, 30Hz, 40Hz, 50Hz

stefan:stefan:

fr~80Hz

fr~80Hz

server

fan

Host 1Host 2

LAN10/100

MBit DifferentialAnalysis

PIII/2.2GHz, GeForce4 4600PIII/500MHz, E&S

Info

rmati

onst

ekn

olo

gi


Results for Test A and Test B

stefan:stefan:

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

1 11

21

31

41

51

61

71

81

91

10

1

11

1

12

1

13

1

14

1

15

1

16

1

17

1

18

1

19

1

20

1

10Hz

20Hz

30Hz

40Hz

50Hz

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

1 11

21

31

41

51

61

71

81

91

10

1

11

1

12

1

13

1

14

1

15

1

16

1

17

1

18

1

19

1

20

1

10Hz

20Hz

30Hz

40Hz

50Hz

Loca

l host

LAN

Animation rate

Animation rate

Frame number

Frame number

#differing pixels

#differing pixels

Info

rmati

onst

ekn

olo

gi


Results for Test A and Test B

stefan:stefan:

Animation Rate 10Hz 20Hz 30Hz 40Hz 50HzAsync. Frames A (%) 0,5 3,5 3 5,5 4,5Async. Frames B (%) 1,5 1,5 1 2,5 5Async. Frames A 1 7 6 11 9Async. Frames B 3 3 2 5 10

0

1

2

3

4

5

6

10Hz 20Hz 30Hz 40Hz 50Hz

Async. Frames A (%)

Async. Frames B (%)

Info

rmati

onst

ekn

olo

gi


Test C and D : Traffic

va = 90°/sec. const.

stefan:stefan:

20 Hz


Client2800x300

Client1800x300

framegrabber800x600

Snurr.wmv

fr~80Hz

fr~80Hz

server

fan

Host 1Host 2

LAN10/100

MBit DifferentialAnalysis

LAN configuration as in Test B

Increasing the number of shared states

Test C: (10 Hz; 1,10,20,40,60,80,100,120,140,160,180,200)

Test D: (20 Hz; 1,10,20,40,60,80,100,120,140,160,180,200)

PIII/2.2GHz, GeForce4 4600PIII/500MHz, E&S

Info

rmati

onst

ekn

olo

gi


Results for Test C and Test D

stefan:stefan:

Number of nodes 1 10 20 40 60 80 100 120 140 160 180 200Async. Frames C (%) 2 2 0,5 0,5 0,5 2 2 3,5 2,5 2,5 3,5 3Async. Frames D (%) 0,5 2 2 2,5 2,5 1,5 1,5 1,5 1 1 2 2Async. Frames C 4 4 1 1 1 4 4 7 5 5 7 6Async. Frames D 1 4 4 5 5 3 3 3 2 2 4 4

0

0,5

1

1,5

2

2,5

3

3,5

4

1 10 20 40 60 80 100 120 140 160 180 200

Async. Frames C (%)

Async. Frames D (%)

Info

rmati

onst

ekn

olo

gi


Conclusion

Framework allows for building modular complex visualization environments

Flexible to use and combine with existing tools

No need for fancy protocols to maintain adequate frame-sync.

Info

rmati

onst

ekn

olo

gi


The fusion 2D and 3D Interfaces

The AQUARIUM is a 3D environment

How can legacy code be re-used?

Can 2D applications be instantiated

within a 3D environment?

Info

rmati

onst

ekn

olo

gi


Virtual Network Computing

System for sharing frame-buffers/applications

Developed by AT&T Laboratories Cambridge,

1999

Open Source

Remote Frame Buffer Protocol well documented

Servers and clients readily available for Microsoft

Windows, Unix, Linux och MacOs

Bold servers, thin clients

Info

rmati

onst

ekn

olo

gi


2D Virtual Network Computing Client

Info

rmati

onst

ekn

olo

gi


A 3D VNC Client

VNC Server

3D VNC Client 3D VNC Client 3D VNC Client

• Decode RFB protocol

• Administrate local texture buffers

• Handle 3D input and map to 2D

server coordinates

x

y

y

x

x’ y’

Info

rmati

onst

ekn

olo

gi


3D VNC Application

Example in the AQUARIUM

Contextual browsing of auxiliary information on the WEB

Info

rmati

onst

ekn

olo

gi


3D VNC Performance Assessment Benchmarking

Frame rate Delay

Common Interaction Text editing Mouse movement Web browsing Streaming videos

Info

rmati

onst

ekn

olo

gi


Modular Application Development

Background The framework & component design Communication between components

The VASE development framework

Info

rmati

onst

ekn

olo

gi


Background Until now – monolithic application Different but similar applications

Many components reappear in several of the applications

Makes modular development natural Framework that handles this is needed

(VASE) Easy script (XML)

The modular approach enables a parallel development process

Info

rmati

onst

ekn

olo

gi


Middlewares used VRT – Virtual Reality Toolkit

For graphics handling Built on top of OpenGL Implements a scenegraph

STREEP For network communication Shared state repository Supports both TCP and UDP

Info

rmati

onst

ekn

olo

gi


Design of the framework and the basic structure of the plugins

The framework defines the basic structure for the components (plugins)

Implements a main module Each plugin is written as a DLL and

implements a class, which structure is inherited from a base class

Plugins are dynamically linked during execution

Easy to change or create new plugins when new functionality is to be added

Main module

Plugin DLL

Class implementation

Plugin instance

Info

rmati

onst

ekn

olo

gi


The main module The main module (vase.exe) handles:

Parsing of a configuration file (*.vas) Loading and creation of plugin instances Distributes user interaction events Handles message passing Handles shared states

”myfile.vas” <component type=vpiAvatar name=”client>

<translate>0 -2 3</translate>

<rotate>0 -90 0</rotate>

<size>2</size>

</component>Huvudmodul (VASE.exe)

Plugin1 Plugin2 Plugin3

Plugin pool

Plugin DLLs

Plugin DLLs

Info

rmati

onst

ekn

olo

gi


Example configuration file

<vase> <component type="AquaViewer" name="tablet_pc_viewer">

<projector>TABLET_PC_VIEW <width>207</width> <height>138</height> <front>50</front> <back>100</back> <window_center>0.0 0.0 0.0</window_center> <window_normal>0.0 1.0 0.0</window_normal> <window_up>0 0.0 -1.0</window_up> <view_point>0.0 200.0 0.0</view_point></projector>

<pipe>TABLET_PC_PIPE <projector>TABLET_PC_VIEW</projector> <left>0</left> <bottom>0</bottom> <width>1024</width> <height>768</height></pipe>

</component>

<component type="AquaWhiteboard" name="wtboard"> <translate>0 0 3</translate> <rotate>0 0 -90</rotate> <scale>1 1 1</scale> <bgcolor_r>0</bgcolor_r> <bgcolor_g>0</bgcolor_g> <bgcolor_b>0</bgcolor_b> <show_pointer>0</show_pointer> </component></vase>

Info

rmati

onst

ekn

olo

gi


Client 2Client 1

Communication Local communication

Function calls to the main module

Method calls

Network communication Uses STREEP Through the main module Message passing

Receiving plugin concatinated to the message

Shared states Subscription lists keeps track

of who is interested in what

Main module

Plugin1 Plugin2

Plugin1

Shared state repository

Main moduleMain module

Plugin2

Info

rmati

onst

ekn

olo

gi


Example of Aqua components (plugins) AquaController

3D-buttons Sends messages to other plugins

AquaEchelon Draws an echelon graph Listens to parsed Stratmas data Implements an internal XML structure for units

AquaEnvironment Handles the drawing of the

map grid

AquaParser Inuput: Stratmas generated data file Output: Parsed binary file Distributes the parsed data to other plugins

AquaViewer Handles projectors and pipes Tracking devices

AquaWhiteboard Overlay for the map Stores strokes in RT90 coordinates

Info

rmati

onst

ekn

olo

gi


How do we connect the AQUA environment to the rest of the world?

The STRATMAS-Link Modifications to STRATMAS Socket connectivity File connectivity XML encoding

Info

rmati

onst

ekn

olo

gi


The STRATMAS Link- Modifications to STRATMAS

TCP socket communication add in a thread

conf.dat (link to) ID number for each Unit

Info

rmati

onst

ekn

olo

gi


The STRATMAS Link- Socket connectivity

Connection on an hostname and port specified in conf.dat

XML data is sent in fixed sized blocks over the socket connection

Communication is one way

Info

rmati

onst

ekn

olo

gi


The STRATMAS Link- File connectivity

Generation of data.txt on the Mac running STRATMAS Xml encoded (see next slide)

Info

rmati

onst

ekn

olo

gi


The STRATMAS Link- XML for Units

int ID;long superior_ID; // -1 if no superiorint rank; // 0-6 - enum RankTypefloat latitude;float longitude;int purpose; // 0-13 - enum PurposeTypeint vehicle_type; // 0-6 - enum VehType, -1 if no vehicleint unit_type; // 0-8 - enum UnitTypeint condition; // 0-100int health; // 0-2 - enum HealthTypelong my_threat; // Larger value - larger threatfloat sens_range; // Sensor range in km, -1 if no sensorsfloat veh_sens_range; // Vehicle sensor range in km, -1 if no vehiclefloat weapon_range; // Shoot radius in km, -1 if no vehiclelong shots_fired; // Number of shots firedlong fi_injured; // Number of enemies injuredlong fi_killed; // Number of enemies killedfloat vx; // vehicle velocity vector in degrees *lng* per

// iteration, -1 if no vehiclefloat vy; // vehicle velocity vector in degrees *lat* per // iteration, -1 if no vehiclefloat speed_kph; // vehicle speed in kilometers per hour, -1 if no vehiclefloat goal_x; // goal position RT90xfloat goal_y; // goal position RT90y

int flag; // Flag for red, green or blue unit. Values: 0, 2 or 1.

Info

rmati

onst

ekn

olo

gi


The STRATMAS Link- XML for Gridint rows; // Number of rows in the grid.int cols; // Number of columns in the grid.int nc; // Number of cells holding relevant data.int nlayers; // Number of layers. char lname[MAX_GRID_LAYERS][64]; // Layer names. int *index; // Array of the cell-indices corresponding to the data

array indices float *data; // The values for each layer are stored sequencially

after eachother. for red, green or blue unit. Values: 0, 2 or 1.

Info

rmati

onst

ekn

olo

gi


Map View

Topographic context Based on RT90 coordinate reference Mapped upon internal grid

1200000

1500000

1900000

6100000

7700000

15º48’29.8’’Ö

50kmx50kmScale 1:100.000

28 x 64 cells á 25x25 km (1:25000)

0/0

700000/0

700000/16000000/1600000

Info

rmati

onst

ekn

olo

gi


Map View contd. 2 textures

1024x1024texels

1024x1024texels

Map cell25 x 25 km32 x 32 texels1 texel ~ 780 meters

700000 [meters (RT90)]

1600

000

[met

ers

(RT9

0)]

Modeling unit in the virtual environment is 1cm (1:1000000)-> Map (70cmx160cm) fits visionarium

Info

rmati

onst

ekn

olo

gi


Map View summary

Map granularity can be adapdedAt present topography is schematic

High resolution map data can be loaded dynamicallyper map cell (e.g. 1:25000 maps per cell)depending on zoom level

Advantage of RT90 reference frameinternal representation of 3D data always Euklideanorthogonal system eases texture mappinguse lat/long conversion routines by metria

Info

rmati

onst

ekn

olo

gi


Cell Grid Layer Co-located with the RT90 map

Mapps geographically related information upon map

Grid cell contain aggregated data for military units

Grid cell contain substrate information

Grid cell can visualize itself

Grid size and resolution can be chosen arbitrarily

Current limits:

Max cell size: 200x200 km2

=> 32 cells/RT90 map

Min cell size: 3.125 x 3.125 km2

=> 114688 cells/RT90 map

Info

rmati

onst

ekn

olo

gi


Cell Properties

Cell geometry (polygon)

Cell texture

Cell related data

num_red : 0, 1,2,3,4,5 => number of red troups in 5 levels 0 = no red troups

num_blue: 0, 1,2,3,4,5 => number of blue troups in 5 levels 0 = no red troups

num_green: 0, 1,2,3,4,5 => number of green troups in 5 levels 0 = no green troups

age: 0, 1,2 => 0 = very up-to-date information; 1 = older; 2=very old

sensor: 0, 1 => 0 = there is no sensor coverage; 1 = sensor coverage

disease: 0, 1,2 => 0 = no disease; 1 = disease1; 2 = disease2

strenght_red: 0, 1,2,3 => 0 = perfect strenght, 1,2,3 dicreasing moral/strenght

strenght_blue: 0, 1,2,3 => 0 = perfect strenght, 1,2,3 dicreasing moral/strenght

strenght_green:0, 1,2,3 => 0 = perfect strenght, 1,2,3 dicreasing moral/strenght

combat: 0, 1,2 => 0 = no fight; 1 = little fight; 2 = heavy fight

Info

rmati

onst

ekn

olo

gi


Mapping Data from external Sources (e.g. Stratmas)

Continuous data(unit data)

Discrete data(raster data)

. . ... .

Stratmas gridrows="79”cols="36"

CellGrid::SetStratmasGridData

CellGrid::SetStratmasUnitData

Info

rmati

onst

ekn

olo

gi


Visualizing Cell Data

Visual pattern library

Patterns associated with properties (32x32 texels)

Patterns can be combined

Examples:

Cell property Visual pattern(s)

Number of troups (ordinal)

No sensor coverage

Diseases

Troup strength

Combat

Age

Info

rmati

onst

ekn

olo

gi


Visualizing Cell Data contd. Rules, based on actual cell data

- Chosing the correct base-pattern- Applying modifiers- Texture combiner

Examples of precompiled patterns (512x512 texels)

Info

rmati

onst

ekn

olo

gi


Visualizing Cell Data contd.

3 forces (red, green, and blue)

3 disease conditions

3 x 3 x 8 megabye/texture = 75 MB texture memory

This is equivalent to 2304 unique patterns

One more attribute will double the memory requirements

This is quite unlikely there are only 1800 grid cells in the default resolutionnot all cells contain interesting data

Cell pattern compilation at run-time for individual cells

Info

rmati

onst

ekn

olo

gi


Cell Grid Visualization Example

Info

rmati

onst

ekn

olo

gi


Visualization of continuously positioned data – Echelon graphs

Info

rmati

onst

ekn

olo

gi


AquaEchelon in detail Function

Draws tactical information in 3D Gives both a spatial and hierarchical

understanding at a glance Interactable

XML Bottom-up vs. Top-down

Info

rmati

onst

ekn

olo

gi


The XML formats Internal format

Top-down Shows a static situation

Example of the internal XML format<echelon> <unit type="skytte" size="pluton" pos="1550000 6600000 "> <weapon type="grk" force="5" range="7"/> <sensor type="radar" range="20"/> </unit>

<unit type="skytte" size="brigad" pos="1202000 6530000 "> <unit type="skytte" size=“bataljon" pos="1200000 6500000 "> <unit type ="grk" size=“kompani" pos="1720000 6530000 "> </unit> <unit size="tropp" pos="2000000 6500000 "> </unit> </unit> <unit type="skytte" size="grupp" pos="1300000 6600000 "> </unit> </unit> </echelon>

Info

rmati

onst

ekn

olo

gi


The XML formats – cont. Stratmas generated

Bottom-up Dynamic, can play whole scenarios

Example of a Stratmas generated file<stratmas> <units flag="red" count=“2"> <unit delay="0"> 1 -1 6 59.383562 17.027027 0 0 0 0.00 0 0.0 0.0 0.0 0.0 0 0 0 0.0 0.0 0.0

59.383562 17.027027 0 </unit> <unit delay="0"> 2 1 5 59.383562 17.027027 0 0 0 0.00 0 0.0 0.0 0.0 0.0 0 0 0 0.0 0.0 0.0

59.383562 17.027027 0 </unit> </units></stratmas>

Info

rmati

onst

ekn

olo

gi


Symbol construction Previously manually generated bitmaps

Now 3D-models of the basic shapes

Symbol construction

. . ..

. .

Info

rmati

onst

ekn

olo

gi


Construction of the symbols Make 3D-models of the basic shapes Merge several of these into a complete unit

Easy to alter the apperance of the symbols by changing the basic models

3D instead of 2D

Info

rmati

onst

ekn

olo

gi


Overlays

Info

rmati

onst

ekn

olo

gi


Overlays

Info

rmati

onst

ekn

olo

gi


Overlays Storage data structure

A document... • Has one or several pages...

• A page has none or several strokes

Rendering in a bitmap texture Strokes are rendered in their correct RT90 context

File format Text/Ascii based format.

Info

rmati

onst

ekn

olo

gi


Interaction in with the AQUA environment

Demo...

Date post:	07-Jan-2016
Category:	Documents
Upload:	robbin
View:	38 times
Download:	3 times

Visual Communication in the AQUA environment

Documents