38
Equalizer Programming Guide http://www.equalizergraphics.com/documents/Developer/ProgrammingGuide.pdf Eyescale Software GmbH Version 1.0 for Equalizer 0.4 October 29, 2007
Equalizer Programming Guidehttp://www.equalizergraphics.com/documents/Developer/ProgrammingGuide.pdf
Eyescale Software GmbH
Version 1.0 for Equalizer 0.4
October 29, 2007
Equalizer 0.4 Programming Guide
Contributors
Written by Stefan Eilemann.Engineering contributions by Maxim Makhinya and Jonas Bosch.
Copyright
©2007 Eyescale Software GmbH. All rights reserved. No permission is granted tocopy, distribute, or create derivative works from the contents of this electronic docu-mentation in any manner, in whole or in part, without the prior written permissionof Eyescale Software GmbH.
Trademarks and Attributions
OpenGL is a registered trademark, OpenGL Multipipe is a trademark of SiliconGraphics, Inc. Linux is a registered trademark of Linus Torvalds. Mac OS is atrademark of Apple Inc. CAVELib is a registered trademark of the Universityof Illinois. The CAVE is a registered trademark of the Board of Trustees of theUniversity of Illinois at Chicago. Qt is a registered trademark of Trolltech. Allother trademarks and copyrights herein are the property of their respective owners.
Feedback
If you have comments about the content, accuracy or comprehensibility of thisprogramming guide, please contact [email protected].
Previous Page
The images on the front page show: a terrain rendering application on a six-nodedisplay wall [top left], the eqPly polygonal renderer in a three-sided CAVE [middle],two volume rendering results from eVolve1 [bottom left and middle] and a six-nodesort-last database decomposition with parallel direct-send recomposition2 [bottomright].
1Data sets courtesy of General Electric, USA and AVS, USA2Data set courtesy of Stanford University Computer Graphics Laboratory
Contents
1 Introduction 1
2 Getting Started 12.1 Installing Equalizer and running eqPly . . . . . . . . . . . . . . . . . 12.2 Equalizer Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2.2.1 Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.2 Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.2.3 Render Clients . . . . . . . . . . . . . . . . . . . . . . . . . . 2
3 Hello, World! 2
4 The Programming Interface 34.1 Task Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34.2 Execution Modeal and Thread Safety . . . . . . . . . . . . . . . . . . 34.3 Config . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4.3.1 Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54.3.2 Pipe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54.3.3 Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64.3.4 Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4.4 Compounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64.4.1 Compound Channels . . . . . . . . . . . . . . . . . . . . . . . 64.4.2 Frustum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64.4.3 Compound Classification . . . . . . . . . . . . . . . . . . . . 74.4.4 Decomposition - Attributes . . . . . . . . . . . . . . . . . . . 74.4.5 Recomposition - Frames . . . . . . . . . . . . . . . . . . . . . 7
5 The eqPly polygonal renderer 75.1 The main Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85.2 Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.2.1 Main Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95.2.2 Render Clients . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5.3 Distributed Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . 125.3.1 InitData - a Static Distributed Object . . . . . . . . . . . . . 125.3.2 FrameData - a Versioned Distributed Object . . . . . . . . . 13
5.4 Config . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135.4.1 Initialization and Exit . . . . . . . . . . . . . . . . . . . . . . 135.4.2 Frame Control . . . . . . . . . . . . . . . . . . . . . . . . . . 155.4.3 Event Handling . . . . . . . . . . . . . . . . . . . . . . . . . . 15
5.5 Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165.5.1 Frame Control . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.6 Pipe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.6.1 Initialization and Exit . . . . . . . . . . . . . . . . . . . . . . 195.6.2 Window System . . . . . . . . . . . . . . . . . . . . . . . . . 195.6.3 Carbon/AGL Thread Safety . . . . . . . . . . . . . . . . . . . 205.6.4 Frame Control . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.7 Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205.7.1 Initialization and Exit . . . . . . . . . . . . . . . . . . . . . . 205.7.2 Object Manager . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.8 Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235.8.1 Initialization and Exit . . . . . . . . . . . . . . . . . . . . . . 235.8.2 Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
6 Advanced Features 276.1 Event Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
6.1.1 Threading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.1.2 Initialization and Exit . . . . . . . . . . . . . . . . . . . . . . 286.1.3 Message Pump . . . . . . . . . . . . . . . . . . . . . . . . . . 286.1.4 Event Data Flow . . . . . . . . . . . . . . . . . . . . . . . . . 296.1.5 Custom Events in eqPixelBench . . . . . . . . . . . . . . . . 29
6.2 Image Compositing for Scalable Rendering . . . . . . . . . . . . . . . 306.2.1 Parallel Direct Send Compositing . . . . . . . . . . . . . . . . 316.2.2 Frame, Frame Data and Images . . . . . . . . . . . . . . . . . 316.2.3 Custom Assembly in eVolve . . . . . . . . . . . . . . . . . . . 32
6.3 Head Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Rev Date Changes1.0 Oct 28, 2007 Initial Version for Equalizer 0.4
1 Introduction
Equalizer provides a framework for the development of parallel OpenGL applica-tions. Equalizer-based applications can run from a single shared-memory systemwith one or multiple graphics cards up to large-scale graphics clusters. This Pro-gramming Guide introduces the programming interface, often using the EqualizereqPly example as a guideline.
Equalizer is the next step in the evolution of generic parallel programming in-terfaces for OpenGL-based visualization applications. Existing solutions, such asOpenGL Multipipe SDK, Cavelib and VRJuggler, implement a subset of conceptssimilar to Equalizer. In other areas, e.g., tracking device support, they providemore functionality.
In order to adapt an application for Equalizer, the programmer structures thesource code so that the OpenGL rendering can be executed in parallel, poten-tially using multiple processes for cluster-based execution. Equalizer provides thedomain-specific parallel rendering know-how and abstracts configuration, thread-ing, synchronization, windowing and event handling. It is a ‘GLUT on steroids’,providing parallel and distributed execution, scalable rendering features and fullycustomizable event handling.
If you have any question regarding Equalizer programming, this programmingguide, or other specific problems you encountered, please direct them to the eq-devmailing list3.
2 Getting Started
2.1 Installing Equalizer and running eqPly
Equalizer can be installed by downloading the distribution4 and compiling thesource code. After installing Equalizer, please take a look at the Quickstart Guide5
to get familiar with the capabilities of the eqPly example.Compiling Equalizer is as simple as running make on Linux or building the Equal-
izer Visual Studio 2005 solution on Windows. On Mac OS X 10.4 (Tiger), some pre-requisites have to be installed before running make, as explained in README.Darwin.Mac OS X 10.5 Leopard does have all the prerequisites installed by default.
2.2 Equalizer Processes
The Equalizer architecture is based on a client-server model. The client library ex-poses all functionality discussed in this document to the programmer, and providescommunication between the different Equalizer processes.
2.2.1 Server
Each Equalizer server is responsible for managing one visualization system, i.e., ashared memory system or graphics cluster. It controls and launches the application’srendering clients. Currently, Equalizer only supports one application per server, butit will provide concurrent and efficient multi-application support in future.
3see http://www.equalizergraphics.com/lists.html4http://www.equalizergraphics.com/downloads.html5http://www.equalizergraphics.com/documents/EqualizerGuide.html
1
2.2.2 Application
The application connects to an Equalizer server and receives a configuration. Fur-thermore, the application also provides its render client, which will be controlled bythe server. The application reacts on events, updates its database and controls therendering.
2.2.3 Render Clients
libeq.soApplication
libeq.soApplication
Render Client
libeq.soApplication
Render Client
libeq.soApplication
Render Client
provides controls
drives libeq.so
Equalizer Server
Figure 1: Equalizer Processes
The render client implements the rendering partof an application. Its execution is passive, it hasno main loop and is completely driven by Equal-izer, based on the rendering tasks received fromthe server. The tasks are executed by callingthe appropriate task methods (see Section 4.1)in the correct thread and context. The appli-cation either implements the task methods withapplication-specific code or uses the default meth-ods provided by Equalizer.
The application can also be a rendering client,in which case it can also contribute to the render-ing. If it does not implement any render client-related code, it is reduced to be theapplication’s ‘master’ process without any OpenGL windows and rendering code.
The rendering client can be the same executable as the application, as it is thecase with all provided examples. When it is started as a render client, the Equalizerinitialization routine does not return and takes over the control by calling the renderclient task methods. Complex applications usually implement a separate, light-weight rendering client.
3 Hello, World!
Figure 2: Hello, World!
The eqHello example is aminimal application to il-lustrate the basic princi-ple of an Equalizer appli-cation: The application de-veloper has to implementthe rendering method Chan-nel::frameDraw, similar to theglutDisplayFunc in GLUT ap-plications. It can be run as astand-alone application fromthe command line.
The eqHello redraw func-tion renders six rotating, col-ored quads around the origin.The frameDraw method pro-vided by the eq::Channel canbe used as a convience function to setup the frustum and other OpenGL state.After setting up some additional lighting parameter, eqHello rotates the scene andrender the quads using immediate mode:
void Channel : : frameDraw ( const u in t 32 t sp in ){
// se tup OpenGL Sta te
2
eq : : Channel : : frameDraw ( sp in ) ;
const f loat l i gh tPos [ ] = { 0 .0 f , 0 . 0 f , 1 . 0 f , 0 . 0 f } ;g l L i gh t f v ( GL LIGHT0 , GL POSITION, l i gh tPos ) ;
const f loat l ightAmbient [ ] = { 0 .2 f , 0 . 2 f , 0 . 2 f , 1 . 0 f } ;g l L i gh t f v ( GL LIGHT0 , GL AMBIENT, l ightAmbient ) ;
// ro t a t e scene around the o r i g i ng lRota t e f ( static cast< f loat >( sp in ) * 0 .5 f , 1 . 0 f , 0 . 5 f , 0 .25 f ) ;
// render s i x axis−a l i gned co lored quads around the o r i g i n[ . . . ]
}
The eqHello main function sets up the communication with the server, initializesand drives the rendering. The details of this setup are explained in Section 5.
4 The Programming Interface
Working on the parallelization of customer applications in order to scale the displaysize and performance for 3D rendering. Design and development of scalable, parallelgraphics applications, 3D graphics software and hardware consulting.
Equalizer uses a C++ programming interface. The API is minimally invasive,so Equalizer imposes only a minimal, natural execution framework upon the appli-cation. It does not provide a scene graph, or interfere in any other way with theapplication’s rendering code. The restructuring work required for Equalizer is theminimal refactoring needed to parallelize the application for rendering.
Methods called by the application have the form verb[Noun], whereas methodscalled by Equalizer (‘Task Methods’) have the form nounVerb. For example, theapplication calls Config::startFrame to render a new frame, which causes –amongother things– Node::frameStart to be called in all active render clients.
4.1 Task Methods
The application inherits from Equalizer classes and overrides virtual functions to im-plement certain functionality, e.g., the application’s OpenGL rendering in eq::Chan-nel::frameDraw. These task methods are similar in concept to C function callbacks.The eqPly section will discuss the most important task methods. A full list can befound on the website6.
4.2 Execution Modeal and Thread Safety
Using threading correctly in OpenGL-based applications is easy with Equalizer.Equalizer creates one rendering thread for each graphics card. All task methodsfor a pipe, and therefore all OpenGL commands, are executed from this thread.This threading model is the OpenGL ‘threading model’, which maintains a currentcontext for each thread. If structured correctly, the application rarely has to takecare of thread synchonization or protection of shared data.
The main thread is responsible for maintaining the application logic. It reactson user events, updates the data model and requests new frames to be rendered. Itdrives the whole application, as shown in Figure 3.
The rendering threads concurrently render the application’s database. The data-base should be accessed in a read-only fashion during rendering to avoid threading
6see http://www.equalizergraphics.com/documents/design/taskMethods.html
3
problems. This is normally the case, for example all modern scene graphs useread-only render traversals.
Application
init send tasks
trigger new frame
wait for frame finish
handle events
update database
exit
Server Render Clients
init
sendrender tasks
executerender tasks
sync frame finish
send tasks exit
idle processing
Figure 3: Simplified execution model
All rendering threads inthe configuration run asyn-chronously to the applica-tion’s main thread. Depend-ing on the configuration’s la-tency, they can fall n framesbehind the last frame finishedby the application thread. Alatency of one frame is usuallynot perceived by the user, butcan increase rendering perfor-mance substantially.
Rendering threads on a sin-gle node are by default syn-chronized. When a frameis finished, all local render-ing threads are done drawing.Therefore the application cansafely modify the data be-tween the end of a frame andthe beginning of a new frame.Furthermore, only one instance of the application data has to be maintained withina process since all rendering threads are guaranteed to draw the same frame.
This per-node frame synchronization does not inhibit latency across renderingnodes. Furthermore, advanced rendering software which multi-buffers the dynamicparts of the database can disable the per-node frame synchronization, as explainedin Section 5.5.1. Some scene graphs do implement multi-buffered data.
4.3 Config
The eq::Config represents the current configuration of the application. The con-figuration is the session in which all render clients are registered. A configurationconsists of the description of the rendering resources and the usage description forthese resources.
The rendering resources are represented in a hierarchical tree structure whichcorresponds to the physical and logical resources found in a 3D rendering environ-ment.
The resource usage is configured using a compound tree, which is a hierarchi-cal representation of the rendering decomposition and recomposition across theresources. It is explained in Section 4.4.
Figure 4 shows an example configuration for a four-side CAVE, running on twomachines (nodes) using three graphics cards (pipes) with one window each to renderto the four output channels connected to the projectors for each of the walls. Thecompound description is only used by the server to compute the rendering tasks.The application is not aware of compounds, and does not need to concern itselfwith the parallel rendering logics of a configuration.
For testing and development purposes it is possible to use multiple instancesfor one resource, e.g. to run multiple render client nodes on one computer. Fordeployment, one node and pipe should be used for each computer and graphicscard, respectively.
4
Resource UsageConfigResources
compoundeye [ LEFT RIGHT ]
channel "left"wall { ... }swapbarrier{}
channel "front"wall { ... }swapbarrier{}
channel "floor"wall { ... }swapbarrier{}
channel "right"wall { ... }swapbarrier{}Channel
name "left"viewport {...}
Windowviewport {...}
Pipe
Node Node
Windowviewport {...}
Pipe
Channelname "floor"
Windowviewport {...}
Pipe
Channelname "front"
Channelname "right"viewport {...}
Figure 4: An example configuration
4.3.1 Node
The eq::Node class is the representation of a single computer in a cluster. Oneoperating system process of the render client will be used for each node. Eachconfiguration might also use an application node, in which case the applicationprocess is also used for rendering. All node-specific task methods are executed fromthe main application thread.
4.3.2 Pipe
The eq::Pipe class is the abstraction of a graphics card (GPU). In the current imple-mentation it is also one operating system thread. Non-threaded pipes are supportedfor integrating with thread-unsafe libraries, but have various performance caveats.All pipe, window and channel task methods are executed from the pipe’s thread, orin the case of non-threaded pipes from the main application thread7.
Further versions of Equalizer might introduce threaded windows, where all window-related task methods are executed in a separate operating system thread.
7see http://www.equalizergraphics.com/documents/design/nonthreaded.html
5
4.3.3 Window
The eq::Window class holds a drawable and an OpenGL context. The drawable canbe an on-screen window or an off-screen8 PBuffer or framebuffer object (FBO). Thewindow holds window-system-specific handles to the drawable and context, e.g., anX11 window XID and GLXContext for the glX window system.
4.3.4 Channel
The eq::Channel class is the abstraction of an OpenGL viewport within its parentwindow. It is the entity executing the actual rendering. The channel’s viewportmight be overwritten when it is rendering for another channel during scalable ren-dering.
4.4 Compounds
Usage of the rendering resources is configured using a compound tree. Althoughthe API does not currently expose compounds, the basic design behind compoundsis explained here for a better understanding of the Equalizer architecture. Fur-ther information on the configuration of compounds can be found on the Equalizerwebsite9.
4.4.1 Compound Channels
Each compound has a channel, which is used by the compound to execute therendering tasks. One channel might be used by multiple compounds. Unusedchannels are not instantiated during initialization. The rendering tasks for thechannels are computed by the server and send to the appropriate render clients.
4.4.2 Frustum
projection
wall
distance
position
bottom-rightbottom-left
top-left
HPRFOV
Figure 5: Wall and projection parameters
Compounds have a frus-tum description to define thephysical layout of the displayenvironment.
The frustum description isinherited by the children,therefore the frustum is typi-cally defined on the top-mostcompound.
The frustrum can be speci-fied as a wall or projection de-scription.
A wall is completely de-fined by the bottom-left,bottom-right and top-left co-ordinates relative to the ori-gin.
The projection is definedby the position and head-pitch-roll orientation of the projector, as well as the hori-zontal and vertical field-of-view and distance of the projection wall.
Figure 5 illustrates the wall and projection frustum parameters.
8Off-screen drawables are not implemented yet, but can be created by the application and usedwith Equalizer
9see http://www.equalizergraphics.com/documents/design/compounds.html
6
4.4.3 Compound Classification
The channels of the leaf compounds in the compound tree are designated as sourcechannels. The top-most channel in the tree is the destination channel. Only sourcechannels execute rendering task. All channels in a compound tree work for thedestination channel. The destination channel defines the 2D pixel viewport renderedby all leaf compounds. The destination channel and pixel viewport can not beoverridden by child compounds.
4.4.4 Decomposition - Attributes
Compounds have attributes which configure the decomposition of the destination’schannel viewport, frustum and database. A viewport decomposes the destinationchannel and frustum in screen space. A range tells the application to render a partof its database, and an eye rendering pass can selectively render different stereopasses. Setting one or multiple attributes causes the parent’s view to be decomposedaccordingly. Attributes are cumulative, that is, intermediate compound attributesaffect and therefore decompose the rendering of all their children.
4.4.5 Recomposition - Frames
Compounds use output and input frames to configure the recomposition of theresulting pixel data from the source channels. An output frame connects to an inputframe of the same name. The selected frame buffer data is transported from theoutput channel to the input channel. The assembly routine of the input channelwill block on the availability of the output frame. This composition process isextensively described in Section 6.2.
5 The eqPly polygonal renderer
In this section the source code of eqPly is explained in detail, and relevant designdecisions, caveats and other remarks are raised.
The eqPly example is shipped with the Equalizer distribution and serves as areference implementation of an Equalizer-based application of medium complexity.It focuses on the example usage of core Equalizer features, not on rendering featuresor visual quality.
configInitframeDraw
Channel
connectServerdisconnectServer
Client
chooseConfigreleaseConfig
Server
initexitstartFramefinishFramesendEventhandleEventshandleEvent
ConfigconfigInitconfigExitframeStartframeFinish
NodeconfigInitconfigExitframeStartframeFinish
PipeconfigInitconfigExitframeStartframeFinishmakeCurrentswapBuffers
WindowconfigInitconfigExitframeStartframeFinishframeClearframeDrawframeReadbackframeAssemble
Channel
*
1*
1
*1
*1*
1
*1
runApplication
initexitstartFramehandleEvent
Config
configInitconfigExit
Node configInitconfigExitframeStart
Pipe
configInitconfigExit
Window
commitsyncgetVersion
setInstanceDatagetInstanceDataapplyInstanceData
setDeltaDatapackunpack
Object
getInstanceDataapplyInstanceData
InitData
FrameData
namespace eq
namespace eqPly
namespace eqNet
Figure 6: UML diagram of significant Equalizer and eqPly classes and task methods.
All classes in the example are in the eqPly namespace to avoid type name ambi-guities, in particular for the Window class which is frequently used as a type in the
7
global namespace by windowing systems. Figure 6 shows how the most importantEqualizer classes are used through inheritance by the eqPly example.
The eqPly classes fall into two categories: Subclasses of the rendering entitiesintroduced in Section 4.3, and two other classes for distributing data. The functionand typical usage for each of the rendering entities is discussed in this section.
The distributed data classes are helper classes based on eqNet::Object. Theyillustrate the typical usage of distributed objects for static as well as dynamic,frame-specific data.
5.1 The main Function
The main function starts off with parsing the command line into the LocalInitDatadata structure. A part of it, the base class InitData, will be distributed to all renderclient nodes. The command line parsing is done by the LocalInitData class, whichis discussed in Section 5.3.1:int main ( int argc , char** argv ){
// 1 . parse argumentseqPly : : Loca l In i tData in i tData ;in i tData . parseArguments ( argc , argv ) ;
The second step is to initialize the Equalizer library. The initialization function ofEqualizer also parses the command line, which is used to set certain default valuesbased on Equalizer-specific options10, e.g., the default server address. Furthermore,a NodeFactory is provided. The EQERROR macro, and its counterparts EQWARN,EQINFO and EQVERB allow selective debugging outputs with various logging levels:
// 2 . Equa l i z e r i n i t i a l i z a t i o nNodeFactory nodeFactory ;i f ( ! eq : : i n i t ( argc , argv , &nodeFactory ) ){
EQERROR << ” Equa l i z e r i n i t f a i l e d ” << endl ;return EXIT FAILURE;
}
The node factory is used by Equalizer to create the object instances of the con-figured rendering entities. Each of the classes inherits from the same type providedby Equalizer in the eq namespace. The provided eq::NodeFactory base class in-stantiates ’plain’ Equalizer objects, thus making it possible to selectively subclassindividual entity types, as it is done by eqHello. For each rendering resources usedin the configuration, one C++ object will be created during initialization:class NodeFactory : public eq : : NodeFactory{public :
virtual eq : : Conf ig * c r ea t eCon f i g ( ) { return new eqPly : : Conf ig ; }virtual eq : : Node* createNode ( ) { return new eqPly : : Node ; }virtual eq : : Pipe* c r ea t eP ipe ( ) { return new eqPly : : Pipe ; }virtual eq : : Window* createWindow ( ) { return new eqPly : : Window ; }virtual eq : : Channel* createChannel ( ) { return new eqPly : : Channel ; }
} ;
The third step is to create an instance of the application and to initialize itlocally. The application is an eq::Client, which in turn is an eqNet::Node. Theunderlying network layer in Equalizer is a peer-to-peer network of eqNet::Nodes.The application programmer does not usually have to be aware of the classes in theeqNet namespace, but both the eq::Client and the server are eqNet::Nodes.
The local initialization of a node creates a local listening socket, which allows theeq::Client to communicate over the network with other nodes, such as the serverand the rendering clients.10Equalizer-specific options always start with - -eq-
8
// 3 . i n i t i a l i z a t i o n o f l o c a l c l i e n t nodeRefPtr< eqPly : : App l i ca t ion > c l i e n t = new eqPly : : App l i ca t ion ( in i tData ) ;i f ( ! c l i e n t −>i n i t L o c a l ( argc , argv ) ){
EQERROR << ”Can ’ t i n i t c l i e n t ” << endl ;eq : : e x i t ( ) ;return EXIT FAILURE;
}
Finally everything is set up, and the eqPly application is executed:
// 4 . run c l i e n tconst int r e t = c l i e n t −>run ( ) ;
After the application has finished, it is de-initialized and the main function re-turns:
// 5 . c leanup and e x i tc l i e n t −>ex i tLo ca l ( ) ;c l i e n t = 0 ;
eq : : e x i t ( ) ;return r e t ;
}
5.2 Application
In the case of eqPly, the application is also the render client. The eqPly executablehas three run-time behaviours:
1. Application: The executable started by the user, the controlling entity inthe rendering session.
2. Auto-launched render client: The typical render client, started by theserver. The server starts the executable with special parameters, which causeClient::initLocal to never return. During exit, the server terminates the process.By default, the server starts the render client using ssh.
3. Resident render client: Manually pre-started render client, listening on aspecified port for server commands. This mode is selected using the command-line option –eq-client and potentially –eq-listen <address> and -r11.
5.2.1 Main Loop
The application’s main loop starts by connecting the application to an Equalizerserver. The command line parameter –eq-server explicitly specifies a server address.If no server was specified, Client::connectServer tries first to connect to a server onthe local machine using the default port. If that fails, it will create a server runningwithin the application process with a default one-channel configuration12.
int Appl i ca t ion : : run ( ){
// 1 . connect to s e rve rRefPtr<eq : : Server> s e r v e r = new eq : : Server ;i f ( ! connectServer ( s e r v e r ) ){
EQERROR << ”Can ’ t open s e r v e r ” << endl ;return EXIT FAILURE;
}
11see http://www.equalizergraphics.com/documents/design/residentNodes.html12see http://www.equalizergraphics.com/documents/design/standalone.html
9
The second step is to ask the server for a configuration. The ConfigParams area placeholder for later Equalizer implementations to provide additional hints andinformation to the server for choosing the configuration. The configuration chosenby the server is created locally using NodeFactory::createConfig. Therefore it is oftype eqPly::Config, but the return value is eq::Config, making the cast necessary:
// 2 . choose con f i geq : : ConfigParams configParams ;Conf ig * c on f i g = static cast<Config*>( s e rver−>chooseConf ig ( configParams ) ) ;
i f ( ! c on f i g ){
EQERROR << ”No matching c on f i g on s e r v e r ” << endl ;d i s connec tSe rve r ( s e r v e r ) ;return EXIT FAILURE;
}
Finally it is time to initialize the configuration. For statistics, the time for thisoperation is measured and printed. During initialization the server launches andconnects all render client nodes, and calls the appropriate initialization task meth-ods, as explained in later sections. Config::init does return after all nodes, pipes,windows and channels are initialized. It returns true only if all initialization taskmethods were successful.
The EQLOG macro allows topic-specific logging. The numeric topic values arespecified in the respective log.h header files, and logging for various topics is enabledusing the environment variable EQ LOG TOPICS:
// 3 . i n i t con f i geqBase : : Clock c l o ck ;
con f i g−>s e t In i tData ( i n i tDa ta ) ;i f ( ! con f i g−> i n i t ( ) ){
EQERROR << ”Config i n i t i a l i z a t i o n f a i l e d : ”<< con f i g−>getErrorMessage ( ) << endl ;
s e rve r−>r e l e a s eCon f i g ( c on f i g ) ;d i s connec tSe rve r ( s e r v e r ) ;return EXIT FAILURE;
}
EQLOG( eq : :LOG CUSTOM ) << ”Config i n i t took ” << c l o ck . getTimef ( ) << ” ms”<< endl ;
When the configuration was successfully initialized, the main rendering loop isexecuted. The main loop runs until the user exits the configuration, or when amaximum number of frames has been rendered, when specified by a command-lineargument. The latter is useful for benchmarks. The Clock is reused for measuringthe overall performance. A new frame is started using Config::startFrame and aframe is finished using Config::finishFrame.
When the frame is started, the server computes all rendering tasks and sendsthem to the appropriate render client nodes. The render client nodes dispatch thetasks to the correct node or pipe thread, where they are executed in order of arrival.
Config::finishFrame synchronizes on the completion of the frame current - latency.The latency is specified in the configuration file, and allows several outstandingframes. This allows overlapping execution in the node and pipe threads and mini-mizes idle times.
By default, Config::finishFrame also synchronizes the completion of all local ren-dering tasks for the current frame. This facilitates porting of exising renderingcodes, since the database does not have to be multi-buffered. Applications suchas eqPly, which do not need this per-node frame synchronization can disable it, asexplained in Section 5.5.1.
10
1
3
2
1
11
2 2 2
1
2
33
33
Config Render Threads
(a)
1
3
2
4
1
11
2
221
2
3
3
33
4 44
Config Render Threads
4
(b)
Figure 7: Synchronous and asynchronous execution
Figure 7 shows the exe-cution of (hypothetical) ren-dering tasks without latency(Figure 7(a)) and with alatency of one frame (Fig-ure 7(b)). With eqPly, aspeedup of 15% has been ob-served on a five-node render-ing cluster when using a la-tency of one frame instead ofno latency13. A latency ofone or two frames is normallynot perceived by the user.
When the main render-ing loop has finished, Con-fig::finishAllFrames is called tocatch up with the latency. Itreturns after all outstandingframes have been rendered,and is needed to provide anaccurate measurement of theframerate:
// 4 . run main loopu in t 32 t maxFrames = in i tDa ta . getMaxFrames ( ) ;
c l o ck . r e s e t ( ) ;while ( con f i g−>isRunning ( ) && maxFrames−− ){
con f i g−>startFrame ( ) ;// conf ig−>renderData ( . . . ) ;con f i g−>f in i shFrame ( ) ;
}const u in t 32 t frame = con f ig−>f i n i shAl lFrames ( ) ;const f loat time = c lock . getTimef ( ) ;EQLOG( eq : :LOG CUSTOM ) << ”Rendering took ” << time << ” ms ( ” << frame
<< ” frames @ ” << ( frame / time * 1000 . f )<< ” FPS) ” << endl ;
The remainder of the application code cleans up in the reverse order of initializa-tion. The config is exited, released and the connection to the server is closed:
// 5 . e x i t con f i gc l o ck . r e s e t ( ) ;con f i g−>e x i t ( ) ;EQLOG( eq : :LOG CUSTOM ) << ”Exit took ” << c l o ck . getTimef ( ) << ” ms” <<endl ;
// 6 . c leanup and e x i ts e rve r−>r e l e a s eCon f i g ( c on f i g ) ;i f ( ! d i s connec tSe rve r ( s e r v e r ) )
EQERROR << ” C l i en t : : d i s connec tSe rve r f a i l e d ” << endl ;s e r v e r = 0 ;return EXIT SUCCESS ;
}
5.2.2 Render Clients
In the second and third use case of the eqPly, when the executable is used as arender client, Client::initLocal never returns. Therefore the application’s main loop
13http://www.equalizergraphics.com/scalability.html
11
is never executed. In order to keep the client resident, the eqPly example overridesthe client loop to keep it running beyond one configuration run:
bool Appl i ca t ion : : c l i en tLoop ( ){
i f ( ! i n i tDa ta . i sRe s i d en t ( ) ) // execute only one con f i g runreturn eq : : C l i en t : : c l i en tLoop ( ) ;
// e l s e execute c l i e n t l oops ’ f o r e v e r ’while ( true ) // TODO: implement SIGHUP handler to e x i t ?{
i f ( ! eq : : C l i en t : : c l i en tLoop ( ) )return fa l se ;
EQINFO << ”One con f i g u r a t i on run s u c c e s s f u l l y executed ” << endl ;}return true ;
}
5.3 Distributed Objects
Equalizer provides distributed objects which help implementing data distributionin a cluster environment. The master version of a distributed object is registeredwith a eqNet::Session, which assigns a session-unique identifier to the object. Othernodes can map their instance of the object to this identifier, thus synchronizing theobject’s data with the remotely registered master version.
Distributed objects are created by subclassing from eqNet::Object. Distributedobjects can be static (immutable) or dynamic. Dynamic objects are versioned.
The eqPly example has a static distributed object to provide initial data to allrendering nodes, as well as a versioned object to provide frame-specific data suchas the camera position to the rendering methods.
5.3.1 InitData - a Static Distributed Object
The InitData class holds a couple of parameters needed during initialization. Theseparameters never change during one configuration run, and are therefore static.
On the application side, the class LocalInitData subclasses InitData to providethe command line parsing and to set the default values. The render nodes onlyinstantiate the distributed part in InitData.
A static distributed object either has to provide a pointer and size to its datausing setInstanceData, or it has to implement getInstanceData and applyInstanceData.The first approach can be used if all distributed member variables are stored in onecontiguous block of memory. The second approach is used otherwise.
The InitData class contains a string of variable length. Therefore it uses thesecond approach of manually serializing and de-serializing its data. Serialization ingetInstanceData and de-serialization in applyInstanceData is performed by streamingall member variable to or from the provided data streams. Efficient buffering anddata transport between nodes is implemented in the data streams:
void In i tData : : get InstanceData ( eqNet : : DataOStream& os ){
os << frameDataID << windowSystem << useVBOs << useGLSL << f i l e name ;}
void In i tData : : applyInstanceData ( eqNet : : DataIStream& i s ){
i s >> frameDataID >> windowSystem >> useVBOs >> useGLSL >> f i l e name ;
EQASSERT( frameDataID != EQ ID INVALID ) ;EQINFO << ”New Ini tData in s t ance ” << endl ;
}
12
The data input and output streams perform no type checking on the data. It isthe application’s responsibility to exactly match the order and types of variablesduring serialization and de-serialization.
5.3.2 FrameData - a Versioned Distributed Object
Versioned objects have to override isStatic to return false, which indicates that theyare versioned. The current implementation has the following characteristics:
� Only the master instance of the object is writable, that is, eqNet::Object::com-mit can be called only on the master instance to generate a new version.
� Slave instance versions can only be advanced, that is, eqNet::Object::sync(version ) with a version smaller than the current version will fail.
� Newly mapped slave instance are mapped to the oldest available version.
Upon commit the delta data from the previous version is sent to all mappedslave instances. The data is queued on the remote node, and is applied when theapplication calls sync to synchronize the object to a new version. The sync methodmight block if a version has not been committed or is still in transmission.
In addition to the instance data (de)serialization methods used to map an object,versioned objects may implement pack and unpack to serialize or de-serialize thechanges since the last version.
If the delta data happens to be layed out contiguously in memory, setDeltaDatamight be used. The default implementation of pack and unpack (de)serialize thedelta data or the instance data if no delta data has been specified.
The eqPly frame data is layed out in one anonymous structure in memory. It alsodoes not track changes since it is relatively small in size and changes frequently.Therefore, the instance and delta data are the same and set in the constructor. Thedefault implementation will take care of the distribution of the data:
FrameData ( ){
r e s e t ( ) ;s e t Ins tanceData ( &data , s izeof ( Data ) ) ;EQINFO << ”New FrameData ” << std : : endl ;
}
5.4 Config
The configuration is driving the application’s rendering, that is, it is responsible forupdating the data based on received events, requesting new frames to be renderedand to provide the render clients with the necessary data.
5.4.1 Initialization and Exit
The config initialization happens in parallel, that is, all config initialization tasks aretransmitted by the server at once and their completion is synchronized afterwards.
The tasks are executed by the node and pipe threads in parallel. The parent’sinitialization methods are always executed before any child initialization method.This parallelization is necessary to allow a speedy startup of the configuration onlarge-scale graphics clusters. On the other hand, it means that initialization func-tions are called even if the parent’s initialization has failed.
13
Config::init
EqualizerServer
Pipe Threads
Application
Client Node Processes
NodeFactory::createNode
NodeFactory::createChannel
NodeFactory::createWindow
NodeFactory::createPipe
Node::configInit
Pipe::selectWSPipe::configInit
Window::configInit
Channel::configInit
Figure 8: Config Initialization Sequence
The eqPly::Config class holdsthe master versions of the ini-tialization and frame data.Both objects are registeredwith the eq::Config, which isthe eqNet::Session used forrendering. Equalizer takescare of the session setup andexit in Client::chooseConfigand Client::releaseConfig, re-spectively.
The frame data is regis-tered before the initializationdata, since its identifier istransmitted using the Init-Data. The identifier of theinitialization data is trans-mitted to the render clientnodes using the initID param-eter of eq::Config::init.
Equalizer will pass this identifier to all configInit calls of the respective objects:
bool Config : : i n i t ( ){
// i n i t d i s t r i b u t e d o b j e c t sframeData . data . c o l o r = in i tDa ta . useColor ( ) ;
r e g i s t e rOb j e c t ( & frameData ) ;i n i tDa ta . setFrameDataID ( frameData . getID ( ) ) ;
r e g i s t e rOb j e c t ( & in i tDa ta ) ;
// i n i t con f i grunning = eq : : Conf ig : : i n i t ( i n i tDa ta . getID ( ) ) ;
i f ( ! running )return fa l se ;
If the configuration was initialized correctly, the configuration tries to set up atracking device for head tracking. Equalizer does not provide extensive support fortracking devices, as this is an orthogonal problem to parallel rendering. Trackingdevice support has already been solved by a number of implementations14, whichcan easily be integrated with Equalizer. The example code in eqPly provides areference implementation for the integration of such a tracking library. Section 6.3provides more background on head tracking.
// i n i t t r acke ri f ( ! i n i tDa ta . getTrackerPort ( ) . empty ( ) ){
i f ( ! t r a c k e r . i n i t ( i n i tDa ta . getTrackerPort ( ) ) )EQWARN << ” Fa i l ed to i n i t i a l i s e t r a cke r ” << endl ;
else{
// Set up po s i t i on o f t r ac k ing system in world space// Note : t h i s depends on the phy s i c a l i n s t a l l a t i o n .vmml : : Matr ix4f m( vmml : : Matr ix4f : : IDENTITY ) ;m. s c a l e ( 1 . f , 1 . f , −1. f ) ;//m. x = . 5 ;t r a c k e r . setWorldToEmitter ( m ) ;
m = vmml : : Matr ix4f : : IDENTITY;m. rotateZ ( −M PI 2 ) ;
14VRCO Trackd, VRPN, etc.
14
t r a c k e r . setSensorToObject ( m ) ;EQLOG( eq : :LOG CUSTOM ) << ”Tracker i n i t i a l i s e d ” << endl ;
}}
return true ;}
The exit function of the configuration stops the render clients by calling eq::Con-fig::exit, and then de-registers the initialization and frame data objects with thesession:bool Config : : e x i t ( ){
running = fa l se ;const bool r e t = eq : : Conf ig : : e x i t ( ) ;
i n i tDa ta . setFrameDataID ( EQ ID INVALID ) ;d e r e g i s t e rOb j e c t ( & in i tDa ta ) ;d e r e g i s t e rOb j e c t ( & frameData ) ;
return r e t ;}
5.4.2 Frame Control
The rendering frames are issued by the application. The eqPly::Config only overridesstartFrame in order to update its data before forwarding the start frame request tothe eq::Config.
If a tracker is used, the current head position and orientation is retrieved andpassed to Equalizer, which uses the head matrix together with the wall or projectiondescription to compute the view frustra15.
The camera position is updated and the frame data is commited, which generatesa new version of this object. This version is passed to the rendering callbacks andwill be used by the rendering threads to synchronize the frame data to the statebelonging to the current frame:u in t 32 t Config : : startFrame ( ){
// update head po s i t i oni f ( t r a c k e r . isRunning ( ) ){
t r a c k e r . update ( ) ;const vmml : : Matr ix4f& headMatrix = t r a c k e r . getMatrix ( ) ;setHeadMatrix ( headMatrix ) ;
}
// update databaseframeData . data . r o t a t i on . preRotateX ( −0.001 f * spinX ) ;frameData . data . r o t a t i on . preRotateY ( −0.001 f * spinY ) ;
const u in t 32 t ve r s i on = frameData . commit ( ) ;
return eq : : Conf ig : : startFrame ( ve r s i on ) ;}
5.4.3 Event Handling
Events are sent by the render clients to the application using eq::Config::sendEvent.At the end of the frame, Config::finishFrame calls Config::handleEvents to do theevent handling. The default implementation processes all pending events by callingConfig::handleEvent for each of them.15see http://www.equalizergraphics.com/documents/design/immersive.html
15
For event-driven execution, the application can override Config::handleEvents toblockingly receive events using Config::nextEvent until a new frame has to be ren-dered.
The eqPly example continuously renders new frames. It implements Config::hand-leEvent to provide the various reactions to user input, most importantly cameraupdates based on mouse events. The camera position has to be handled correctlywith respect to latency, and is therefore saved in the frame data:
bool Config : : handleEvent ( const eq : : ConfigEvent* event ){
switch ( event−>type ){
[ . . . ]case eq : : ConfigEvent : : POINTER MOTION:
i f ( event−>pointerMotion . buttons == eq : :PTR BUTTON NONE )return true ;
i f ( event−>pointerMotion . buttons == eq : :PTR BUTTON1 ){
spinX = 0 ;spinY = 0 ;
frameData . data . r o t a t i on . preRotateX (−0.005 f * event−>pointerMotion . dx ) ;
frameData . data . r o t a t i on . preRotateY (−0.005 f * event−>pointerMotion . dy ) ;
}else i f ( event−>pointerMotion . buttons == eq : :PTR BUTTON2 | |
event−>pointerMotion . buttons == ( eq : :PTR BUTTON1 |eq : :PTR BUTTON3 ) )
{frameData . data . t r a n s l a t i o n . z +=
.005 f * event−>pointerMotion . dy ;}else i f ( event−>pointerMotion . buttons == eq : :PTR BUTTON3 ){
frameData . data . t r a n s l a t i o n . x +=.0005 f * event−>pointerMotion . dx ;
frameData . data . t r a n s l a t i o n . y −=.0005 f * event−>pointerMotion . dy ;
}return true ;
default :break ;
}return eq : : Conf ig : : handleEvent ( event ) ;
}
5.5 Node
For each active render client, one eq::Node instance is created on the appropriatemachine. Nodes are only instantiated on their render client processes, i.e., eachprocess should only have one instance of the eq::Node class. The application processmight also have a node class, which is handled in exactly the same way as the renderclient nodes.
During node initialization the init data is mapped to a local instance using theidentifier passed from Config::init. The model is loaded based on the filename in theinitialization data. No pipe, window or channel tasks methods are executed beforeNode::configInit has returned.
bool Node : : c o n f i g I n i t ( const u in t 32 t i n i t ID ){
16
eq : : Conf ig * c on f i g = getConf ig ( ) ;const bool mapped = con f ig−>mapObject ( & in i tData , i n i t ID ) ;EQASSERT( mapped ) ;
const s t r i n g& f i l ename = in i tDa ta . getFi lename ( ) ;EQINFO << ”Loading model ” << f i l ename << endl ;
model = new Model ( ) ;i f ( ! model−>readFromFile ( f i l ename . c s t r ( ) ) ){
EQWARN << ”Can ’ t load model : ” << f i l ename << endl ;delete model ;model = 0 ;
}
return eq : : Node : : c o n f i g I n i t ( i n i t ID ) ;}
The node config exit function deletes the loaded model and unmaps the initial-ization data:
bool Node : : c on f i gEx i t ( ){
delete model ;model = NULL;
eq : : Conf ig * c on f i g = getConf ig ( ) ;con f i g−>unmapObject ( & in i tDa ta ) ;
return eq : : Node : : c on f i gEx i t ( ) ;}
5.5.1 Frame Control
The application has extended control over the task synchronization during a frame.Upon Config::startFrame, Equalizer invokes the frameStart task methods of thevarious entities. The entities unlock all its children by calling startFrame, e.g.,Node::frameStart has to call Node::startFrame in order to unlock the pipe threads.Note that certain startFrame calls, e.g., Window::startFrame, are currently emptysince the synchronization is implicit due to the sequential execution within thethread.
Likewise, Config::finishFrame causes the invokation of the frameFinish task meth-ods. These task methods unlock their parents by calling releaseFrame.
The explicit synchronization of child or parent resources allows the application tooptimize the processing, by performing certain, independent operations when thechild or parent resources are already unlocked.
The default thread synchronization synchronizes all Channel::frameDraw opera-tions on a single node with the node’s main thread. This facilitates porting, sincethe scene database does not have to be multi-buffered as all threads of the nodeprocess render the same frame.
The per-node frame synchronization does not break the asynchronous executionacross rendering nodes. Advanced applications can remove per-node frame synchro-nization.
The per-node frame synchronization is achieved through the startFrame�wait-FrameStarted and releaseFrameLocal�waitFrameLocal synchronization points. Notethat this synchronization is only per-node, different nodes in the cluster still runasynchronously.
The first synchronization point ensures that the application (node) thread isdone modifiying the data, typically during event processing. The pipe threads call
17
Node::waitFrameStarted which blocks until the node calls startFrame in Node::frame-Start.
The second pair ensures that after Config::finishFrame all pipe threads are donerendering the current frame, and therefore do not access shared data. At this pointthe application thread can safely modify scene data. The node’s frameDrawFinishwaits for all local pipes to release the synchronization by calling Pipe::releaseFra-meLocal, which happens by default in Pipe::frameDrawFinish. The frameDrawFinishmethods are called after all Channel::frameDraw of the corresponding thread havebeen executed.
Pipe(s)Application Node
Config::startFrame
Pipe(s)
Node::frameStart
Server
Config::finishFrame
Render ClientApplication
Node::startFrameNode::frameStart
Node::startFrame Pipe::frameStartConfig::beginFrame
Pipe::frameStart
Pipe::startFrame
Node::frameFinishNode::releaseFrame
Pipe::frameFinishPipe::releaseFrame
Node::waitFrameStarted
Node::frameDrawFinishPipe(s)::waitFrameLocal
draw tasks
Pipe::frameDrawFinishPipe::releaseFrameLocal
assemble tasks
Pipe::startFrameNode::waitFrameStarted
Pipe::frameFinishPipe::releaseFrame
draw tasks
Pipe::frameDrawFinishPipe::releaseFrameLocal
assemble tasks
Node::frameFinishNode::releaseFrame
Node::frameDrawFinishPipe(s)::waitFrameLocal
Figure 9: Synchronization of frame tasks
Figure 9 outlines the synchronization for the application, node and pipe classes.Please note that Config::finishFrame does block until the current frame has beenreleased locally and until the frame current - latency has been released by all nodes.The window and channel synchronization are similar and omitted for simplicity.
It is absolutely vital for the execution that Node::startFrame and Node::releaseFra-me are called, respectively. The default implementation of the node task methodsdoes take care of that.
Since eqPly multi-buffers all dynamic data, it can completely remove frame syn-chronization by:
� releasing the local synchronization early in Node::frameStart
� not waiting for the node to start the frame by not calling Node::waitFrameStartedin Pipe::frameStart
� not waiting for the pipe synchronization in Node::frameDrawFinish
Releasing the local synchronization early is done by calling releaseFrameLocal,and Node::frameDrawFinish is overwritten with an empty implementation:void Node : : frameDrawFinish ( const u in t 32 t frameID ,
const u in t 32 t frameNumber ){ /* nop , see frameStart */ }
void Node : : f rameStart ( const u in t 32 t frameID , const u in t 32 t frameNumber ){
startFrame ( frameNumber ) ; // unlock pipe threads
// Don ’ t wai t f o r p ipes to r e l e a s e frame l o c a l l y , sync not needed s ince a l l// dynamic data i s mult i−bu f f e r e dre leaseFrameLoca l ( frameNumber ) ;
}
18
5.6 Pipe
All task methods for a pipe and its children are executed in a separate thread.This approach optimizes usage of the GPU, since all tasks are executed serially andtherefore do not compete for resources or cause OpenGL context switches. Laterversions of Equalizer might introduce threaded windows to allow the parallel andindependent execution of rendering tasks on a single pipe.
5.6.1 Initialization and Exit
Pipe threads are not explicitely synchronized with each other, that is, pipes mightbe rendering different frames at any given time. Therefore frame-specific data hasto be allocated for each pipe thread, which in the eqPly example is the frame data.The frame data is a member variable of the eqPly::Pipe, and is mapped to theidentifier provided by the initialization data. The initialization in eq::Pipe doesthe GPU-specific initialization, which is window-system-dependent. On AGL thedisplay ID is determined, and on glX the display connection is opened.
bool Pipe : : c o n f i g I n i t ( const u in t 32 t i n i t ID ){
const Node* node = static cast<Node*>( getNode ( ) ) ;const In i tData& in i tData = node−>get In i tData ( ) ;const u in t 32 t frameDataID = in i tData . getFrameDataID ( ) ;eq : : Conf ig * c on f i g = getConf ig ( ) ;
const bool mapped = con f ig−>mapObject ( & frameData , frameDataID ) ;EQASSERT( mapped ) ;
return eq : : Pipe : : c o n f i g I n i t ( i n i t ID ) ;}
The config exit function is similar to the config initialization. The frame data isunmapped and GPU-specific data is de-initialized by eq::Config::exit:
bool Pipe : : c on f i gEx i t ( ){
eq : : Conf ig * c on f i g = getConf ig ( ) ;con f i g−>unmapObject ( & frameData ) ;
return eq : : Pipe : : c on f i gEx i t ( ) ;}
5.6.2 Window System
Equalizer supports multiple window system interfaces, at the moment glX/X11,WGL and AGL/Carbon. Some operating systems, and therefore some Equalizerversions, support multiple window systems concurrently16.
Each pipe might use a different window system for rendering, which is determinedbefore Pipe::configInit by Pipe::selectWindowSystem. The default implementation ofselectWindowSystem loops over all window systems and returns the first supportedwindow system, determined by using supportsWindowSystem.
The eqPly examples allows selecting the window system using a command lineoption. Therefore the implementation of selectWindowSystem is overwritten andreturns the specified window system, if supported:
eq : : WindowSystem Pipe : : selectWindowSystem ( ) const{
const Node* node = static cast<Node*>( getNode ( ) ) ;const In i tData& in i tData = node−>get In i tData ( ) ;const eq : : WindowSystem ws = in i tData . getWindowSystem ( ) ;
16see http://www.equalizergraphics.com/compatibility.html
19
i f ( ws == eq : :WINDOW SYSTEM NONE )return eq : : Pipe : : selectWindowSystem ( ) ;
i f ( ! supportsWindowSystem ( ws ) ){
EQWARN << ”Window system ” << ws<< ” not supported , us ing d e f au l t window system” << endl ;
return eq : : Pipe : : selectWindowSystem ( ) ;}
return ws ;}
5.6.3 Carbon/AGL Thread Safety
Parts of the Carbon API used for window and event handling in the AGL windowsystem are not thread safe. The application has to call eq::Global::enterCarbon be-fore any thread-unsafe Carbon call, and eq::Global::leaveCarbon afterwards. Thesefunctions should be used only during window initialization and exit, not during ren-dering. For various reasons enterCarbon might block up to 50 milliseconds. Carboncalls in the window event handling routine Window::processEvent are thread-safe,since the global carbon lock is set in this method. Please contact the Equalizerdeveloper mailing list if you need to use Carbon calls on a per-frame basis.
5.6.4 Frame Control
All task methods for a given frame of the pipe, window and channel entities be-longing to the thread are executed in one block, starting with Pipe::frameStart andfinished by Pipe::finishFrame. The frame start callback is therefore the natural placeto update all frame-specific data to the version belonging to the frame.
In eqPly, the version of the only frame-specific object FrameData is passed as theper-frame id from Config::startFrame to the frame task methods. The pipe uses thisversion to update its instance of the frame data to the current version, and thenunlocks its child entities by calling startFrame:
void Pipe : : f rameStart ( const u in t 32 t frameID , const u in t 32 t frameNumber ){
// don ’ t wai t f o r node to s t a r t frame , l o c a l sync not needed// node−>waitFrameStarted ( frameNumber ) ;frameData . sync ( frameID ) ;
startFrame ( frameNumber ) ;}
5.7 Window
The Equalizer window holds an OpenGL drawable and a rendering context. Whenusing the default window initialization functions, all windows of a pipe share theOpenGL context. This allows reuse of OpenGL objects such as display lists andtextures between all windows of one pipe.
The window class is the natural place for the application to maintain all dataspecific to the OpenGL context.
5.7.1 Initialization and Exit
The initialization sequence uses multiple, overrideable task methods. The main taskmethod configInit executes a ‘child’ task method to create the drawable and context.The child task method depends on the window system of the pipe. The default im-plementations of configInitGLX, configInitWGL or configInitAGL create an on-screen
20
window using OS-specific methods. If the OpenGL drawable and context were cre-ated successfully, configInit calls configInitGL, which performs the generic OpenGLstate initialization. The default implementation sets up some typical OpenGL state,e.g., it enables the depth test.
Start
configInit
window-system
configInitGLX configInitWGL configInitAGL
GLX WGL AGL
failed?
configInitGL
success
failed?
failure
no
yes
yes
no
Figure 10: Window Initialization
Figure 10 shows a flow chart of the window ini-tialization. The colored functions are task meth-ods and can be replaced by application-specificimplementations.
The window-system specific initialization takesinto account various attributes set in the con-figuration file. Attributes include the size ofvarious frame buffer attachments (color, alpha,depth, stencil) as well as other framebuffer prop-erties, such as quad-buffered stereo, doublebuffer-ing, fullscreen mode and window decorations.Some of the attributes, such as stereo, double-buffer and stencil can be set to eq::AUTO, inwhich case Equalizer will test for their availabilityand enable them if possible.
The eqPly window initialization function firstcalls eq::Window::configInit to use the generic win-dow setup. If that was successful, it initializes astate object and an overlay logo:
bool Window : : c o n f i g I n i t ( const u in t 32 t i n i t ID ){
i f ( ! eq : : Window : : c o n f i g I n i t ( i n i t ID ) )return fa l se ;
eq : : Pipe* pipe = getPipe ( ) ;Window* f irstWindow = static cast< Window* >( pipe−>getWindow( 0 ) ) ;
EQASSERT( ! s t a t e ) ;
i f ( f irstWindow == this ){
s t a t e = new VertexBuf f e rState ( getGLFunctions ( ) ) ;loadLogo ( ) ;
const Node* node = static cast< const Node* >( getNode ( ) ) ;const In i tData& in i tData = node−>get In i tData ( ) ;
i f ( in i tData . useVBOs ( ) ){
const eq : : GLFunctions* g lFunct ions = getGLFunctions ( ) ;// Check i f a l l VBO funcs a va i l a b l e , e l s e l e ave DISPLAY LIST MODE oni f ( g lFunct ions−>hasGenBuffers ( ) && glFunct ions−>hasBindBuffer ( ) &&
glFunct ions−>hasBufferData ( ) && glFunct ions−>hasDe l e t eBu f f e r s ( ) ){
s t a t e−>setRenderMode ( mesh : :BUFFER OBJECT MODE ) ;EQINFO << ”VBO render ing enabled ” << endl ;
}else
EQWARN << ”VBO func t i on po i n t e r s miss ing , us ing d i sp l ay l i s t s ”<< endl ;
}}else{
s t a t e = firstWindow−> s t a t e ;l ogoTexture = firstWindow−> l ogoTexture ;l o g oS i z e = firstWindow−> l o g oS i z e ;
21
}
i f ( ! s t a t e ) // happens i f f i r s t window f a i l e d to i n i t i a l i z ereturn fa l se ;
return true ;}
The state object is used to handle the creation of OpenGL objects in a mul-tipipe, multi-threaded execution environment. It uses an object manager, whichis described in detail in Section 5.7.2. It is used in conjunction with a reference-counting smart-pointer here, since it is potentially ‘owned’ by multiple windows atthe same time.
The logo texture is loaded from the file system and bound to a texture ID usedlater by the channel for rendering. A code listing is ommitted, since the codeconsists of standard OpenGL calls and is not Equalizer-specific.
The window exit function de-allocates all OpenGL objects when the state objectis about to be disposed. The object manager does not delete the object in its de-structor, since it does not know if an OpenGL context is still current. Additionally,eq::Window::configExit is called to destroy the drawable and context:
bool Window : : c on f i gEx i t ( ){
i f ( s t a t e . i sVa l i d ( ) && s ta t e−>getRefCount ( ) == 1 )s t a t e−>d e l e t eA l l ( ) ;
s t a t e = 0 ;return eq : : Window : : c on f i gEx i t ( ) ;
}
5.7.2 Object Manager
The object manager is not strictly a part of the window. It is mentioned here sincethe eqPly window uses an object manager.
The state object in eqPly gathers all rendering state, which includes an objectmanager for OpenGL object allocation.
The object manager (OM) is a utility class and can be used to manage OpenGLobjects across shared contexts. Typically one OM is used for each set of sharedcontexts and spawns all contexts of a single GPU17.
The OM is a template class. The template type is the key used to identify objects.The same key is used by all contexts to get the OpenGL name of an object. In eqPly,a key of type const void * is used. The rendering code uses the address of the dataitem to be rendered as the key to obtain the associated OpenGL object.
All objects managed by the OM are reference counted. If an application releasesthe objects properly, they are automatically de-allocated. It is also possible tomanually manage de-allocation of objects, which might be more convenient in somecases.
Currently, support for display lists, VBO’s, textures and shaders is implemented.For each object, the following functions are available:
supportsObjects() returns true if the usage for this particular type of objects issupported. For objects available in OpenGL 1.1 or earlier, this function is notimplemented.
getObject( key ) returns the object associated with the given key, or FAILED.Increases the reference count of existing objects.
17http://www.equalizergraphics.com/documents/design/objectManager.html
22
newObject( key ) allocates a new object for the given key. Returns FAILED if theobject already exists or if the allocation failed. Sets the reference count of anewly created object to one.
obtainObject( key ) convenience function which gets or obtains the object associ-ated with the given key. Returns FAILED only if the object allocation failed.
releaseObject( key | name ) decreases the reference count and deletes the objectif the reference count reaches zero.
deleteObject( key | name ) manually deletes the object. To be used if referencecounting is not used.
5.8 Channel
The channel is the heart of the application in that it contains the actual renderingcode. The channel is used to perform the various rendering operations for thecompounds.
5.8.1 Initialization and Exit
During channel initialization, the near and far planes are set to reasonable valuesto contain the whole model. During rendering, the near and far planes are adjusteddynamically to the current model position:
bool Channel : : c o n f i g I n i t ( const u in t 32 t i n i t ID ){
setNearFar ( 0 . 1 f , 10 .0 f ) ;return true ;
}
5.8.2 Rendering
The central rendering routine is Channel::frameDraw. This routine contains theapplication’s OpenGL rendering code, which is being rendered using the contextualinformation provided by Equalizer. As most of the other task methods, frameDrawis called in parallel by Equalizer on all pipe threads in the configuration. Thereforethe rendering must not write to shared data, which is the case for all major scenegraph implementations.
In eqPly, the OpenGL context is first set up using various apply conveniencemethods from the base Equalizer channel class. Each of the apply methods uses thecorresponding get method(s) and then calls the appropriate OpenGL function(s).It is also possible to just query the values from Equalizer using the get methods,and use them to set up the OpenGL state appropriatly, for example by passing theparameters to the renderer used by the application.
For example, the implementation for eq::Channel::applyBuffer does set up the cor-rect rendering buffer and color mask, which depends on the current eye pass andpossible anaglyphic stereo parameters:
void eq : : Channel : : app lyBuf fe r ( ){
glReadBuf fer ( getReadBuffer ( ) ) ;glDrawBuffer ( getDrawBuffer ( ) ) ;
const ColorMask& colorMask = getDrawBufferMask ( ) ;glColorMask ( colorMask . red , colorMask . green , colorMask . blue , true ) ;
}
23
The contextual information has to be used in order to render the view as expectedby Equalizer. Failure to use certain information will result in incorrect renderingfor some or all configurations. The channel render context consist of:
Buffer The OpenGL read and draw buffer as well as color mask. These parametersare influenced by the current eye pass, eye separation and anaglyphic stereosettings.
Viewport The two-dimensional pixel viewport restricting the rendering area withinthe channel. For correct operations, both glViewport and glScissor have to beused. The pixel viewport is influenced by the destination channel’s viewportdefinition and compound viewports set for sort-first/2D decompositions.
Frustum The same frustum parameters as defined by glFrustum. Typically thefrustum used to set up the OpenGL projection matrix. The frustum is influ-enced by the destination channel’s view definition, compound viewports, headmatrix and the current eye pass.
Head Transformation A transformation matrix positioning the frustum. This istypically an identity matrix and is used for off-axis frustra in immersive ren-dering. It is normally used to set up the ‘view’ part of the modelview matrix,before static light sources are defined.
Range A one-dimensional range with the interval [0..1]. This parameter is optionaland should be used by the application to render only the appropriate subsetof its data. It is influenced by the compound range attribute.
The frameDraw method in eqPly calls the frameDraw method from the parent class,the Equalizer channel. The default frameDraw method uses the apply conveniencefunctions to setup the OpenGL state for all render context information, with theexception of the range which will be used later during rendering:void eq : : Channel : : frameDraw ( const u in t 32 t frameID ){
app lyBuf f e r ( ) ;applyViewport ( ) ;
glMatrixMode ( GL PROJECTION ) ;g lLoadIdent i ty ( ) ;applyFrustum ( ) ;
glMatrixMode ( GL MODELVIEW ) ;g lLoadIdent i ty ( ) ;applyHeadTransform ( ) ;
}
void Channel : : frameDraw ( const u in t 32 t frameID ){
// Setup OpenGL s t a t eeq : : Channel : : frameDraw ( frameID ) ;
Figure 11: Destination view of anDB compound
After the basic view setup, a directional lightis configured, and the model is positioned usingthe camera parameters from the frame data. Thecamera parameters are transported using the theframe data to ensure that all channels render agiven frame using the same position.
Furthermore, a white color is set in case themodel does not contain color information, or thecolor information is not used. In sort-last render-ing, eqPly uses a different color for each channel
24
to illustrate the database decomposition, as shown in Figure 11. The Equalizerchannel provides a method to obtain a random, but unique color for all channelsin the configuration. This color is determined by the server to ensure uniquenessacross all channels of the configuration:
g lL i gh t f v ( GL LIGHT0 , GL POSITION, l i g h tP o s i t i o n ) ;g lL i gh t f v ( GL LIGHT0 , GL AMBIENT, l ightAmbient ) ;g lL i gh t f v ( GL LIGHT0 , GL DIFFUSE, l i g h tD i f f u s e ) ;g lL i gh t f v ( GL LIGHT0 , GL SPECULAR, l i g h tSp e cu l a r ) ;
g lMa t e r i a l f v ( GL FRONT, GL AMBIENT, materialAmbient ) ;g lMa t e r i a l f v ( GL FRONT, GL DIFFUSE, ma t e r i a lD i f f u s e ) ;g lMa t e r i a l f v ( GL FRONT, GL SPECULAR, mate r i a lSpecu la r ) ;g lMa t e r i a l i ( GL FRONT, GL SHININESS , mat e r i a l Sh i n i n e s s ) ;
const Pipe* pipe = static cast<Pipe*>( getPipe ( ) ) ;const FrameData& frameData = pipe−>getFrameData ( ) ;
g lT r an s l a t e f ( frameData . data . t r a n s l a t i o n . x ,frameData . data . t r a n s l a t i o n . y ,frameData . data . t r a n s l a t i o n . z ) ;
g lMultMatr ixf ( frameData . data . r o t a t i on . ml ) ;
Node* node = (Node*) getNode ( ) ;const Model* model = node−>getModel ( ) ;const eq : : Range& range = getRange ( ) ;
i f ( ! range . i s F u l l ( ) ) // Color DB−patches{
const vmml : : Vector3ub co l o r = getUniqueColor ( ) ;g lColor3ub ( c o l o r . r , c o l o r . g , c o l o r . b ) ;
}else i f ( ! frameData . data . c o l o r | | ( model && ! model−>hasColors ( ) ) ){
g lCo l o r 3 f ( 1 . 0 f , 1 . 0 f , 1 . 0 f ) ;}
Finally the model, which has been loaded by the node, is rendered. If the modelwas not loaded during node initialization, a quad is drawn in its place:
i f ( model ){
drawModel ( model ) ;}else{
g lCo l o r 3 f ( 1 . f , 1 . f , 0 . f ) ;g lNormal3f ( 0 . f , −1. f , 0 . f ) ;g lBeg in ( GL TRIANGLE STRIP ) ;g lVe r t ex3 f ( . 25 f , 0 . f , . 25 f ) ;g lVe r t ex3 f ( . 25 f , 0 . f , −.25 f ) ;g lVe r t ex3 f ( −.25 f , 0 . f , . 25 f ) ;g lVe r t ex3 f ( −.25 f , 0 . f , −.25 f ) ;glEnd ( ) ;
}}
In order to draw the model, a helper class for view frustum culling is set up usingthe view frustum from Equalizer and the camera position from the frame data. Thefrustum helper computes the six frustum planes from the projection and modelViewmatrices. During rendering, the bounding spheres of the model are tested againstthese planes to determine the visibility with the frustum.
Furthermore, the render state from the window and the database range from thechannel is obtained. The render state manages display list or VBO allocation:void Channel : : drawModel ( const Model* model ){
25
Window* window = static cast<Window*>( getWindow ( ) ) ;mesh : : Ver texBuf f e rState& s t a t e = window−>ge tSta t e ( ) ;
const Pipe* pipe = static cast<Pipe*>( getPipe ( ) ) ;const FrameData& frameData = pipe−>getFrameData ( ) ;
const eq : : Range& range = getRange ( ) ;vmml : : FrustumCul ler f c u l l e r ;
s t a t e . s e tCo l o r s ( frameData . data . c o l o r &&range . i s F u l l ( ) &&model−>hasColors ( ) ) ;
in i tFrustum ( cu l l e r , model−>getBoundingSphere ( ) ) ;
model−>beginRendering ( s t a t e ) ;
The model data is spatially organized in an 3-dimensional kD-tree18 for efficientview frustum culling. When the model is loaded by Node::configInit, it is prepro-cessed into the kD-tree and each node of the tree gets a database range assigned.The root node has the range [0, 1], its left child [0, 0.5] and its right child [0.5, 1],and so on for all nodes in the tree. The preprocessed model is saved in a binaryformat for accelerating subsequent use.
setup render state
add root node to candidates
candidates empty?
get candidate
no
fully visible?
no
partially visible?
no
cleanup render stateyes
yesrender candidate
yesadd children to
candidates
Start
Stop
fully in range?yes
has children?yes
no
in range?yes
no
no
render candidate
Figure 12: Main Rendering Loop
The rendering loop main-tains a list of candidates torender, which initially con-tains the root node. Eachcandidate of this list is testedfor full visiblity against thefrustum and range, and ren-dered if visible. It is droppedif it is fully invisible or fullyout of range. If it is partiallyvisible or partially in range,the children of the node areadded to the candidate list.Figure 12 shows a flow chartof the rendering algorithm,which performs efficient viewfrustum and range culling.
The actual rendering usesdisplay lists or vertex bufferobjects. These OpenGL ob-jects are allocated using theobject manager. The render-ing is done by the leaf nodes,which are small enough tostore the vertex indices in a short value for optimal performance with VBO’s. Theleaf nodes reuse the objects stored in the object manager, or create and set up newobjects if it was not yet set up. Since one object manager is used per thread (pipe),this allows a thread-safe sharing of the compiled display lists or VBO’s across allwindows of a pipe.
The main rendering loop in eqPly looks like this:
model−>beginRendering ( s t a t e ) ;
// s t a r t with root nodevector< const VertexBufferBase * > cand idate s ;
18See also http://en.wikipedia.org/wiki/Kd-tree
26
cand idate s . push back ( model ) ;
while ( ! cand idate s . empty ( ) ){
const VertexBufferBase * treeNode = cand idate s . back ( ) ;cand idate s . pop back ( ) ;
// comple te l y out o f range checki f ( treeNode−>getRange ( ) [ 0 ] >= range . end | |
treeNode−>getRange ( ) [ 1 ] < range . s t a r t )continue ;
// bounding sphere view frustum c u l l i n gswitch ( c u l l e r . t e s tSphere ( treeNode−>getBoundingSphere ( ) ) ){
case vmml : : VISIBILITY FULL :// i f f u l l y v i s i b l e and f u l l y in range , render i ti f ( treeNode−>getRange ( ) [ 0 ] >= range . s t a r t &&
treeNode−>getRange ( ) [ 1 ] < range . end ){
treeNode−>render ( s t a t e ) ;break ;
}// p a r t i a l range , f a l l through to p a r t i a l v i s i b i l i t y
case vmml : : VISIBILITY PARTIAL :{
const VertexBufferBase * l e f t = treeNode−>ge tLe f t ( ) ;const VertexBufferBase * r i g h t = treeNode−>getRight ( ) ;
i f ( ! l e f t && ! r i g h t ){
i f ( treeNode−>getRange ( ) [ 0 ] >= range . s t a r t )treeNode−>render ( s t a t e ) ;
// e l s e drop , to be drawn by ’ prev ious ’ channel}else{
i f ( l e f t )cand idate s . push back ( l e f t ) ;
i f ( r i g h t )cand idate s . push back ( r i g h t ) ;
}break ;
}case vmml : : VISIBILITY NONE :
// do nothingbreak ;
}}
model−>endRendering ( s t a t e ) ;}
6 Advanced Features
This section discusses some additional important features not covered by the pre-vious eqPly section. Where possible code examples from the Equalizer distributionare used.
6.1 Event Handling
Event handling requires a lot of flexibility. On one hand, the implementation differsslightly for each operating and window system due to conceptual differences in the
27
OS-specific implementation. On the other hand, each application and widget sethas its own model on how events are to be handled. Therefore, event handling inEqualizer is customizable at any stage of the processing, to the extreme of makingit possible to disable all event-related code in Equalizer. In this aspect, Equalizersubstantially differs from GLUT, which imposes an event model and hides most ofthe event handling in glutMainLoop. More information on event handling can befound on the Equalizer website19.
The default implementation provides a convenient, easily accessible event frame-work, while allowing all necessary customizations. It gathers all events in themain thread of the application, so that the developer only has to implement Con-fig::processEvent to update its data based on the pre-processed, generic keyboardand mouse events. It is very easy to use and similar to an GLUT-based implemen-tation.
6.1.1 Threading
Where possible events are received and processed by a separate per-node eventthread to allow asynchronous20 event handling. Currently an event thread is onlyused by the X11/glX window system. WGL receives and processes the events fromthe pipe threads that created the windows. AGL receives the events from applicationor node main thread. Whenever the term event thread is used, it refers to thethread receiving the event, such as a per-node thread for glX, the pipe thread forWGL and the main thread for AGL.
6.1.2 Initialization and Exit
config initpipe init
get event thread
ApplicationPipe Thread(s)
Event Threadadd pipe to
event threadopen pipe
window init
add window to event thread
listen for window events
Figure 13: Event Handling Ini-tialization
During window and pipe initialization the eventhandling is set up. For both entities, initEvent-Handler is called to register the pipe or windowwith an event handler. This method may be over-written to use a custom event handler, or to notinstall an event handler at all and so disable eventhandling. Likewise, exitEventHandler is called tode-initialize event handling.
An event handler consists of two parts: thegeneric base class providing the interface andgeneric functions, and the window-system-specificpart providing the actual implementation.
Event handling is initialized whenever a newwindow-system-specific pipe or window handle is set. First, exitEventHandler iscalled to de-initialize event handling for the old handle (if set), and then initEvent-Handler is called for the new handle.
AGL and glX use an event handler singleton, whereas WGL uses one event handlerper window. Pipe event handling is only used for glX, where one Display connectionis created and used to subscribe to window events.
6.1.3 Message Pump
For the WGL and AGL window systems, it is required to manually receive and dis-patch (‘pump’) events. On WGL, this has to happen on each thread with windows,whereas on AGL it has to happen only on the main thread. By default, Equalizerpumps these events automatically for the application.
19see http://www.equalizergraphics.com/documents/design/eventHandling.html20with respect to the rendering
28
The methods Client::useMessagePump and Pipe::useMessagePump can be over-ridden to return false to disable this behaviour for their respective threads. Onnon-threaded pipes, Pipe::useMessagePump is not called.
If the application disables message pumping in Equalizer, it has to make sure theevents are pumped externally, as it often done within external widget sets such asQt.
6.1.4 Event Data Flow
OS Event
EventHandler
WindowEvent
ConfigEvent
OS Event
Window::processEvent
Config::sendEvent
Event Thread App Thread
Config::handleEvents
Config::handleEvent
EventQueue
ConfigEvent
Config::finishFrame
Figure 14: Event Processing
Events are received by an event handler. Theevent handler finds the eq::Window associated tothe event. It then creates a generic WindowEvent,which holds important event data in a windowsystem-independent format. The original eventis attached to the generic window event.
The event handler then passes the windowevent to Window::processEvent, which is respon-sible for either handling the event locally, or fortranslating it into a generic ConfigEvent. The con-fig events are send to the application thread usingConfig::sendEvent.
If the event was processed by processEvent,the function has to return true. If false is re-turned, the event will be passed to a previouslyinstalled, window-system-specific event handlingfunction. The default implementation of Win-dow::processEvent passes most events on to theapplication.
Events sent using Config::sendEvent are queued in the application thread. After aframe has been finished, Config::finishFrame calls Config::handleEvents. The defaultimplementation of this method provides non-blocking event processing, that is, itcalls Config::handleEvent for each queued event. By overriding handleEvents, event-driven execution can be implemented.
Later Equalizer versions will introduce Pipe::processEvent and PipeEvent to com-municate pipe-specific events, e.g., monitor resolution changes.
6.1.5 Custom Events in eqPixelBench
The eqPixelBench example is a benchmark program to measure the pixel transferrates from and to the framebuffer of all channels within a configuration. It usescustom config events to send the gathered data to the application. It is muchsimpler than the eqPly example since it does not provide any useful rendering oruser interaction.
The rendering routine of eqPixelBench in Channel::frameDraw loops through anumber of pixel formats and types. For each of them, it measures the time toreadback and assemble a full-channel image. The format, type, size and time isrecorded in a config event, which is sent to the application.
The ConfigEvent derives from the eq::ConfigEvent structure and has the followingdefinition:
struct ConfigEvent : public eq : : ConfigEvent{public :
enum Type{
READBACK = eq : : ConfigEvent : : USER,
29
ASSEMBLE} ;
ConfigEvent ( ){
s i z e = s izeof ( ConfigEvent ) ;}
// channel name i s in user event datachar formatType [ 6 4 ] ;vmml : : Vector2 i area ;f loat msec ;
} ;
The Config::sendEvent method transmits an eq::ConfigEvent or derived class tothe application. The ConfigEvent has to be a C-type structure, and its size memberhas to be set to the full size of the event to be transmitted. Each event has a typewhich is used to identify it by the config processing function.
User-defined types start at eq::ConfigEvent::USER, and the member variable Con-figEvent::user can be used to store EQ USER EVENT SIZE21 bytes. In this space,the channel’s name is stored. Additional variables are used to transport the pixelformat and type, the size and the time it took for rendering.
On the application end, Config::handleEvent uses the ostream operator for thederived config event to output these events in a nicely formatted way:
std : : ostream& operator << ( std : : ostream& os , const ConfigEvent* event ) ;. . .bool Config : : handleEvent ( const eq : : ConfigEvent* event ){
switch ( event−>type ){
case ConfigEvent : :READBACK:case ConfigEvent : :ASSEMBLE:
cout << static cast< const ConfigEvent* >( event ) << endl ;return true ;
default :return eq : : Conf ig : : handleEvent ( event ) ;
}}
6.2 Image Compositing for Scalable Rendering
Two task methods are responsible for collecting and compositing the result imageduring scalable rendering. Scalable rendering is a use case of parallel rendering,when multiple channels are contributing to a single view.
The source channels producing one or more outputFrames use Channel::frame-Readback to read the pixel data from the frame buffer. The channels receiving oneor multiple inputFrames use Channel::frameAssemble to assemble the pixel data intothe framebuffer. Equalizer takes care of the network transport of frame buffer databetween nodes, if needed.
Normally the programmer does not need to interfere with the image composit-ing. Changes are sometimes required at a high level, for example to order theinput frames or to optimize the readback. The following sections provide a detaileddescription of the image compositing API in Equalizer.
21currently 32 bytes
30
6.2.1 Parallel Direct Send Compositing
In order to provide a motivation for the design of the image compositing API, thedirect send parallel compositing algorithm is introduced in this section.
read
-bac
k
send/receivecolor + depth
asse
mbl
e/re
ad-b
ack
asse
mbl
e
source 1(destination) source 2 source 3
send/receivecolor
Figure 15: Parallel Direct Send Compositing
The main idea for di-rect send is to parallelizethe costly recomposition fordatabase (sort-last) decom-position. With each ad-ditional source channel, theamount of pixel data tobe composited grows linearly.When using the simple ap-proach of compositing allframes on the destinationchannel, this channel quicklybecomes the bottleneck in thesystem. Direct send dis-tributes this workload evenlyacross all source channels,and thereby keeps the com-positing work per channelconstant.
In direct send compositing,each rendering channel is alsoresponsible for the sort-lastcomposition22 of one screen-space tile. He receives the framebuffer pixels for histile from all the other channels. The size of one tile decreases linearly with thenumber of source channels, which keeps the total amount of pixel data per channelconstant.
After performing the sort-last compositing, the color information is transferredto the destination channel, similarly to an 2D (sort-first) compound. The amountof pixel data for this part of the compositing pipeline also approaches a constantvalue, i.e., the full frame buffer.
Figure 15 illustrates this algorithm for three channels. The Equalizer websitecontains a presentation explaining this algorithm23.
The following operations have to be possible in order to perform this algorithm:
� Selection of frame buffer attachments: color and/or depth
� Restricting the read back area to a part of the rendered area
� Positioning the pixel data correctly on the receiving channels
Furthermore it should be possible for the application to implement a read backof only the relevant region of interest, that is, the 2D area of the framebuffer actu-ally updated during rendering. This optimization will be fully supported by laterversions of Equalizer.
6.2.2 Frame, Frame Data and Images
An eq::Frame references an eq::FrameData. The frame data is the object connectingoutput with input frames. Output and input frames with the same name withinthe same compound tree will reference the same frame data.22depth-based for polygonal data or sorted-blend for volume data23http://www.equalizergraphics.com/documents/EGPGV07.pdf
31
The frame data is a holder for images and additional information, such as outputframe attributes and pixel data availability.
An eq::Image holds a two-dimensional snapshot of the framebuffer and can containcolor and/or depth information.
The frame synchronization through the frame data allows the input frame towait for the pixel data to become ready, which is signalled by the output frameafter readback.
Furthermore, the frame data transports the inherited range of the output frame’scompound. The range can be used to compute the assembly order of multiple inputframes, e.g., for sorted-blend compositing in volume rendering applications.
Readback and assemble operations on frames and images are designed to beasynchronous. They have a start and finish method for both readback and assem-ble to allow the initiation and synchronization of the operation. Currently, onlysynchronous readback and assembly using glReadPixels and glDrawPixels is imple-mented in the respective start method of the image. Later versions of Equalizerwill implement asynchronous pixel transfers.
Frame
FrameOffset
FrameData
FrameDataPVP
Image
Image
ImagePVP
Figure 16: Hierarchy of assembly classes
The offset of input and out-put frames characterizes theposition of the frame datawith respect to the frame-buffer, that is, the window’slower-left corner. For outputframes this is the position ofthe channel with respect tothe window.
For output frames, theframe data’s pixel viewport isthe area of the frame buffer toread back. It will transportthe offset from the source tothe destination channel, thatis, the frame data pixel view-port for input frames positionthe pixel data on the destina-tion. This has the effect thata partial framebuffer readback will end up in the same place in the destinationchannels.
The image pixel viewport signifies the region of interest that will be read back.The default readback operation reads back one image using the full pixel viewportof the frame data.
Figure 16 illustrates the relationship between frames, frame data and images.
6.2.3 Custom Assembly in eVolve
The eVolve example is a scalable volume renderer. It uses 3D texture-based volumerendering, where the 3D texture is intersected by view-aligned slices. The slices arerendered back-to-front and blended together to produce the final image, as shownin Figure 1724.
When using 2D (sort-first) or stereo decompositions, no special programming isneeded to achieve good scalability, as eVolve is mostly fill-limited and thereforescales nicely in this mode.
24Volume Data Set courtesy of: SFB-382 of the German Research Council (DFG)
32
ViewDirection
Figure 17: Blending Slices in 3D-Texture-based VolumeRendering
The full power of scalable volume rendering ishowever in DB (sort-last) compounds, where thefull volume is divided into separate bricks. Eachof the bricks is rendered like a separate volume.For recomposition, the RGBA frame buffer dataresulting from these render passes then has to beassembled correctly.
Conceptually, the individual volume bricks ofeach of the source channels produces pixel datawhich can be handled like one big ’slice’ throughthe full texture. Therefore they have to be blen-ded back-to-front in the same way as the sliceplanes during rendering.
DB compounds have the advantage of scalingany part of the volume rendering pipeline: tex-ture and main memory (smaller bricks for eachchannel), fill rate (less samples per channel) andIO bandwidth for time-dependent data (less data per time step and channel). Sincethe amount of texture memory needed for each node decreases linearly, they makeit possible to render data sets which are not feasible to visualize with any otherapproach.
Back t
o Fron
t
Compo
siting
Zmodel
Zviewα
nearplane
(a)
1 2 3 4
(b)
Figure 18: Back-to-Front Compositing for Orthogonal and Perspective View Frustra
For recomposition, the 2D frame buffer contents are blended together to form aseamless picture. For correct blending, the frames are ordered in the same back-to-front order as the slices used for rendering, and use the same blending parameters.Simplified, the frame buffer images are ‘thick’ slices which are ‘rendered’ by writingtheir content to the destination frame buffer using the correct order.
For orthographic rendering, determining the compositing order of the input framesis trivial. The screen-space orientation of the volume bricks determines the orderin which they have to be composited. The bricks in eVolve are created by slic-ing the volume along one dimension. Therefore the range of the resulting framebuffer images, together with the sorting order, is used to arrange the frames duringcompositing. Figure 18(a) shows this composition for one view.
Finding the correct assembly order for perspective frustra is more complex. Theperspective distortion invalidates a simple orientation criteria like the one used for
33
orthographic frustra. For the view and frustum setup shown in Figure 18(b)25 thecorrect compositing order is 4-3-1-2 or 1-4-3-2.
Figure 19: Result of Figure 18(b)
In order to compute the assembly order, eVolveuses the angle between the origin→slice vectorand the near plane, as shown in Figure 18(b).When the angle becomes greater than 90°, thecompositing order of the remaining frames has tobe changed. The result image of this compositionnaturally looks the same as the volume renderingwould when rendered on a single channel. Fig-ure 19 shows the result of the composition fromFigure 18(b).
The assembly algorithm described in this sec-tion also works with parallel compositing algo-rithms such as direct-send.
6.3 Head Tracking
The eqPly example contains rudimentary support for head tracking, in order toshow how head tracking can be integrated with Equalizer. Support for a widerange of tracking devices is not within the scope of Equalizer. Other open sourceand commercial implementations cover this functionality sufficiently and can easilybe integrated with Equalizer.
x
wall definition from configuration file
near plane
far plane
(a)
x
z
(b)
x
z
(c)
Figure 20: Monoscopic, Stereoscopic and Tracked frustra
Figure 20(a) illustrates a monoscopic view frustum. The viewer is positioned atthe origin of the global coordinate system, and the frustum is completely symmetric.This is the typical view frustum for non-stereoscopic applications.
In stereo rendering, the scene is rendered twice, with the two frustra ’moved’ bythe distance between the eyes, as shown in Figure 20(b).
In immersive visualization, the observer is tracked in and the view frustra areadapted to the viewer’s position and orientation, as shown in Figure 20(c). Thetransformation origin→ viewer is set by the application using Config::setHeadMatrix,which is used by the server to compute the frustra. The resulting off-axis frustraare positioned using the channel’s head transformation.
25Volume Data Set courtesy of: AVS, USA
34
