PRINCIPLES OF COMPUTER GRAPHICS – UNITN 0809

Project – Live Video in World Wind

Jovan Stevovic

2

Summary Project description ...................................................................................................................................................3

Proposed architecture ..............................................................................................................................................5

Class diagram ........................................................................................................................................................5

Visibility of images ................................................................................................................................................7

Quality of images ..................................................................................................................................................8

Managing Images............................................................................................................................................... 10

Supported Webcams ......................................................................................................................................... 12

Data representation .............................................................................................................................................. 14

Sequence diagrams................................................................................................................................................ 15

Conclusions ............................................................................................................................................................ 18

3

Project description

The goal of the project is to show videos from different sources like webcams or local video on the Nasa World

Wind. The actual Java SDK version 1.4 provides a lot of features for drawing objects, images, icons, annotations

etc. on the World Window but there are not features for displaying videos or images that can changes their

aspect during the time. The main task that was elaborated within the project was trying to find the best way in

terms of performance and quality, to shows images those changes continuously. The architecture of the

current implementation of World Wind Java SDK is shown in Figure 1.

Figure 1: The WWJ architecture

The SDK provides a simple way to add own implementations and compositions of objects on the Globe using

Layers. A Layer (Figure 2.) contains Renderable objects and provides the possibility to personalize the way in

which they will be displayed. A Renderable object provides some functions that permit to draw and move

objects on the Globe. These objects can be stored locally, into the cache or can be on remote servers.

Figure 2: Layer architecture

Within the project was developed mainly a new Layer that contains Videos that has an image source and

displays them using a new Renderable objects. The new Renderable object has the main characteristic that the

4

texture can be updated periodically. The proposed architecture of the new developed Layer is shown in Figure

3.

Figure 3: Developed architecture

The Layer is responsible also for schedule the update of Renderables in a way that permits the normal usage of

World Wind. This means that the fluid and fast navigation cannot be compromised. Adding video sources to

the “system” introduces some problems in terms of efficiency. For the data streams managing in Java was used

the JMF 2.1.1e library. The classic and simplest way to implement a reproduction of some source of

information is using a Player. Player in JMF is thread based data stream managers that provides a lot of

functions for acquisition, processing and delivery of media. That media is transformed in Images, scaled to the

preferred quality, rotated and delivered to the Renderable object. The Renderable object reloads the texture

and then draws it on the Globe. All these tasks are scheduled into the Layer level using threads in order to

avoid blocking of the navigation and considering the trade-off between the quality and the speed of updating

of images and the usability of the application.

In the next chapters will be illustrated in details the entire architecture of the application.

5

Proposed architecture

Class diagram

The architecture of the developed application is shown into the class diagram in Figure 4. The diagram shows

all implemented classes. Some of them are utility classes for data management and some of them extends or

implements existent WWJ classes and are the core of the application. The WWJ’s classes that are used for

developing are blue painted. The 2 also blue painted Packages are used for data streams and player

management.

Figure 4 : Class Diagram

The main class is the ApplicationMain that extends a JApplet and starts the World Wind adding the VideoLayer

to it and adding also a menu for managing webcam types and searching webcams.

6

The VideoLayer class extends AbstractLayer and is responsible for updating DynamicSurfaceImage classes

getting images from VideoObjects. The update task is done at variable intervals depending on how much time

it spends on getting and updating images. More time is spent for updating images larger are intervals. This is

done for usability reasons.

The DynamicSurfaceImage extends Renderable and is very similar to the SurfaceImage class with methods for

updating the source of the image.

VideoObject is the class that represents a video instance. It has a VideoSource, a VideoInfo and a VideoMenu.

The VideoSource class is an interface that specifies essential functions that one specialized video source must

implement. In this project are implemented two types of video sources. A player for local video tapes and a

Webcam parser that is able to capture images from some types of remote webcams.

VideoInfo contains all information about the webcam and the place where it’s situated.

VideoMenu implements the menu of an Image and uses two types of icons. UserFacingIcon for buttons and

GlobeAnnotation for the annotation that shows info about the place. It uses an IconRenderer for rendering the

icons.

PointsManager has the main function to manage points that represents image corners. As we can see later the

user are able to drag and drop image corners in order to create a perspective view of the image. This class

move points, recalculate the sector dimensions of that image and supply some other utility functions.

SearchUtil is used to perform search activities on webcams.

ImagesUtils provides all functions needed for managing images before rendering them.

XMLUtils provides all functions needed for loading information from configuration files and from webcams

folder.

7

Visibility of images

As described in previous paragraphs VideoLayer manages the updating process of DynamicSurfaceImages.

Some considerations can be done about the visibility of images and their updating process in order to improve

performances of the application. If we are viewing an image the others that are not in the View shouldn’t be

updated and then can be paused. An example is shown in Figure 5.

Figure 5: Videos not in View are paused.

Another consideration is that if the image is so sloped then the user probably is not interested in that image

and in any case doesn’t see the image very well, so the updating can be paused. Example in Figure 6.

Figure 6: A sloped view of an image

If there are objects between the view eye and the image then the updating of the image has to be stopped.

This is useful in particular in mountain landscape. Example in figure 7.

8

Figure 7: An intersection detection

Quality of images

Captured images can have variable dimensions and sometimes can be very big having dimensions greater than

800x600. During the navigation, the user can be close or far from the image like in Figure 8. If the user is so

close then the image has to be showed in the best possible quality, but if the user is far from the image then

the dimension can be reduced in order to improve the efficiency of reloading texture.

Figure 8: Far and near view of the same image

9

In Figure 9 are shown some statistics about how much time requires the rendering process at varying the

image dimension. We can see that the required time for rendering images decrease drastically on decreasing

image dimension.

Figure 9: Time to render images of different quality

All these checking functions are done at every updating cycle and permits to have a bigger number of

simultaneously played videos. As described before the update cycles are delayed more if the updating and

rendering requires more time. The update and render time are proportional to the number of played videos. In

Figure 10 are shown some statistics.

Figure 10: Time to reload a number of images

Required time for reloading is stored in the Statistics class and at every updating cycle the VideoLayer ask to

Statistics class an estimated time to “wait” for the next upload. This improves the usability of the application.

10

Managing Images

The application is an Applet that has a menu panel on the left side as we can see in Figure 11.

Figure 11: The application

In the box 1 are listed all types of webcams in witch can be categorized. The user can remove or add types or

clicking on the name can list all webcams of this type.

Box 2 has two buttons that permits to add a webcam or a country to the list of countries. Clicking on the add

webcam button, the boxes 2, 3, 4 will be replaced by the insert form as we can see in next figure.

The box 3 permits to search webcam on the Globe. The user can select the text to search, to search only in

active webcams or also into them that are disabled by users in the box 1. The user can also select the country

in which to search and to include the URL into the possible result.

In Box 4 are shown search results. The user can go to the webcam, to modify it or also to delete it.

If the user click to Add Webcam then will be shown a form that contains some essential information like the

URL and some other meta-information. The user has to pick two angles of the rectangle that represents the

dimension and position of the image. The form and rotation can be changed successively.

11

Figure 12: Insert mode

The meta information that user has to insert are the country, the place name, the altitude of the place, the

type of the webcam and a little description.

When the webcam is playing the user can open the menu of the image double-clicking on it. The menu

contains a button that permits the deletion of the webcam, a button that pause the execution, a settings

button and an information button that shows an annotation with meta-information. On the right side there are

2 buttons that permits to increase or decrease the quality of the image. Some webcams deliver images with

very high quality but some other don’t. The user can try to increase then the quality reaching the maximum.

12

Figure 13: Image Menu

Clicking on the setting button will be shown a second menu that allow user to modify meta-information and

also the form of the image. The possibility to manage the form of the image permits to adapt them to the

terrain conformation. There are two types of buttons. The angles one permits to drag only a corner. The

middles one permit to move both angles at the corners simultaneously. Activating the drag and drop menu is

activated also the BasicDragger that permits to move the entire image. Using the menu on the MenuPanel the

user can also rotate the image by some degree. This is done rotating buttons circularly by the centre using

trigonometric functions.

Figure 14: Image modification menu

Supported Webcams

Nowadays there are a lot of webcams that are published on internet and are accessible in various ways. The

simplest way is by URL. Webcams captures images and saves them on some accessible directory. Then the user

access on that directory by URL and display the image with a browser. If the user wants to see the “next frame”

has to reload the browser. Some other webcams are showed by Java Applets and sometimes users can manage

13

some parameters like direction, zoom etc. There are webcams that can be accessed directly bi IP. These cams

are called IP-Cams and can be accessed from any client that support streams of media. In this project are used

Jipcam libraries that provide access to “Axis” (http://www.axis.com/products/video/camera/index.htm)

webcams. These webcams are the most popular and has the advantage that provides a stream of data that can

be captured easily. The supported models are:

• Axis205

• Axis206M

• Axis207 accessing mode: “/axis-cgi/mjpg/video.cgi” or “/axis-cgi/jpg/image.cgi”

• Axis207M

• Axis207MW

• Axis2100 accessing mode: “/cgi-bin/image.cgi” or “/mjpg/mjpg/video.mjpg”

• Axis2120

• Axis2130

Google provides a way to search pages with some substring into the URL, so if we search for example

• inurl: /axis-cgi/mjpg/video.cgi

• intitle:Live view - / - AXIS

• inurl:axis-cgi/jpg

• inurl:view/indexFrame.shtml

• inurl:view/view.shtml

• inurl:view/index.shtml

• allintitle: Network Camera NetworkCamera

• intitle:liveapplet

• intitle:"WJ-NT104 Main"

• intitle:"snc-rz30 home"

the result will be a list of available cameras that the user can add to the project.

There are also a lot of websites that classifies and provides lists of webcams by manufacturer and what these

webcams shows. The list of the most popular cites:

• http://www.camhacker.com/

• http://www.123cam.com/category-webcam-Axis.html

• http://www.ttrix.com/ipvisionpubcameras.html

• http://www.snow-cams.com/

14

Data representation

Videos and Webcams information are stored on local memory as xml files and have a structure like in Figure

15. The main element is a Video. It has some elements that describe video characteristics and a sub-element

that contains information about the place that the video shows.

Figure 15: The XML representation of a video

All XML files that contain video information are divided into folders by country and then by continent in a way

that simplifies the insertion and modification process.

Figure 16: The webcam classification

15

Sequence diagrams In Figure 4 is shown the class diagram of the entire project. In the next figures are shown the interactions that

all classes have during the starting of the application and the normal usage. In Figure 17 is shown the sequence

diagram of the starting process of the application.

Figure 17: Sequence Diagram of starting the application

The main applet creates the VideoLayer and start it. The VIdeoLayer initialize renderables invoking the load

function from XMLUtil class that load data from xml files described before. When the VideoLayer obtains the

renderables then starts the Thread that updates them continuously. At each cycle the layer asks for the

current frame captured by VideoSources that are inside VideoObjects. Once obtained the elaborated frame as

BufferedImage then substitute it to the DynamicSurfaceImage source of texture and render it. At each cycle

VideoLayer measure the time for getting and updating images and add it to the Statistics class. After that the

VideoLayer get the new estimated time to sleep in order to have a fluid navigation through the World Wind.

A lot of features are situated into the MenuPanel on the left side of the Canvas. User can view webcams of a

desiderated type, search them, add, cancel, modify etc. These functionalities are provided mainly by XML and

Search utilities that implements algorithms to access and modify the xml files using JDom. In the next Sequence

Diagram is shown the search process.

16

Figure 18: Sequence Diagram of the Search operation

User fills the form for the searching function and then press the Search button. If the user choose to search

only from active webcams then the search function ask for the list of active webcams to the VideoLayer. Once

returned the function pass all parameters of the search form to the Search utility that read xml files and return

results to the MenuPanel. Then the results are viewed on the results panel.

Another important functionality that is provided by menu buttons is the Drag and Drop function of image

angles that permits to modify the form of the image as described in previous chapter. There are two ways to

access the Drag and Drop menu; from image menu of from MenuPanel on the left side of the Canvas.

17

Figure 19: Drag and Drop of image points

The user activates the menu in one of the two ways (in this figure using the MenuPanel). The Points are

showed by VideoMenu class that has an IconRender. User picks points (angle or middle points) and moves

them. The new positions of picked icons are passed to the PointsManager that recalculate the new positions of

other points, the new sector dimension and the new image proportions in that sector of the image. These

proportions as described before are passed to the Video Source and at each update cycle the image will be

elaborated using these points. The update of image is not immediately showed because the update is done by

the VideoLayer using Statistic functions.

18

Conclusions WWJ don’t provide features for reloading images dynamically and the texture loading process is expensive in

terms of CPU time. Invoking a reloading function at each incoming frame will compromise the usage of the

application. The update function must be invoked a number of times smaller than the frame rate of videos in

order to have a good usability.

The process of getting and elaborating images is also expensive. Every Player has an internal Thread that wait

for events from the source. If the source has a high frame rate and we want to elaborate each frame then this

will probably consume all CPU time that we have.

So we have to analyze very well the trade-off between the quality of images, the frequency of reloading and

the usability of the application. The way in which the trade-off is implemented into the application consists in

two steps. One is given by Statistics evaluation of consumed time to reload and elaborate images in order to

obtain a good time to wait at each reloading cycle. The second one is given by asynchronous invocation of

functions that returns Images from sources. For doing that is used the Threaded Queue of tasks that WWJ

implements. In this way the image is got asynchronously and the reloading is maybe delayed but the navigation

is fluid.

The totality of CPU time is spent by the acquisition of images and then by the showing process on the Globe.

During the work was considered the possibility to use the GPU computing power (CUDA for NVIDIA graphics

cards) but there is not sufficient amount of arithmetic operations for justify the use of this technique that can

compromise the compatibility of the application with other operating systems and machines.

Webcams has an angle of view variable. Some cameras show a street angle and some other shows an entire

mountain. The quality of WW image is not very high so if we want to put a webcam that shows a street we

don’t have a way to show it on real street and real position. In the next few examples are shown webcams on

Google Earth images quality. In these examples we can see how can be useful and nice the combination of high

quality of images and webcam geolocalization.

Figure 20:Seebruke Selling, Germany

19

Figure 21: A bridge in Newcastle, UK

Figure 22: Stuttgart City airport, Germany