VOMegaPlot Efficient Plotting of Large VOTable Datasets.

transcript

VOMegaPlot

Efficient Plotting of Large VOTable Datasets

VOPlot

VOPlot is a tool for visualizing astronomical data that is available in the VOTable format.

VOPlot reads the xml file in order to load entire data into memory and then processes it to draw various types of plots.

This approach of loading the entire data into memory cannot be used for VOTable files that are very large.

Approach for VOMegaPlot VOMegaPlot preprocesses the xml file to

create intermediate files which are subsequently used for plotting.

Entire data is divided into fixed sized blocks and individual blocks are loaded into memory thus reducing the memory requirement.

The number of intermediate files created is equal to the number of columns present in the xml file.

Pre-processing operationCreation of array blocks

Col 1 Col m …….

File 2

…….

Original xml file with m columns and n rows

Block 1

File 1

Block 2

Block k

Block 1

Block 2

Block k

File m

Block 1

Block 2

Block k

Intermediate files on disk

Algorithm for drawing a scatter plot

1) Input the columns to be plotted, say A vs. B

2) Load a set of corresponding blocks for both columns, A and B.

3) Take corresponding data elements from both the blocks and plot them.

4) After plotting all the points, discard the blocks.

5) If there exist more blocks of data repeat step 2, else stop.

Advantages

The complexity for plotting is to O(2n) where n is the no. of rows. This complexity is independent of the no. of columns in the xml file.

If the user has to plot only a subset of data (as in case of zoom operation) then there exists another set of files which can be used for this purpose.

Dealing with subset of data

Data for every column is stored in an indexed fashion.

This helps in accessing the subset of data without having to go through the entire set of data.

As a result, operations like zoom become much faster.

Pre-processing operationCreation of tree blocks

Col 1 Col m …….

Indexed File for col 1

Indexed File for col 2

…….

Original xml file with m columns and n rows

Intermediate files with indexed data

0.1–0.3

0.3–0.6

Indexed File for col m

Pre-processing operationCreation of tree blocks (contd)

0-20 20-40 40-60

20-30 30-40

30-35 35-40

Indexed file for a column

Results

Tycho Tycho-2 UCAC2

Data size 1 million rows and 56 columns

2.5 million rows and 32 columns

48.3 million rows and 9 columns

Pre-processing time

18 minutes 30 minutes 3 hours 26 minutes

Plotting time for scatter plot

9 seconds 22 seconds 5 minutes 46 seconds

Features of VOMegaPlot

Scatter Plot with zoom, reverse axis and logged axis

Projection Plot Density Plot Histogram

Scatter PlotTycho-1 catalogue ( RA vs. Vmag)

Density PlotTycho-1 catalogue ( RA vs. Vmag)

Density PlotTycho-2 catalogue ( DEC vs RA)

Scatter Plot UCAC2 Catalogue (2m_J vs. U2Rmag)

Density PlotUCAC2 Catalogue (2m_J vs. U2Rmag)

Future Enhancements

Support for reading data stored in binary format

Block level compression while creating intermediate files

Client Server version

References VOTable

http://www.ivoa.net/Documents/latest/VOT.html

VOPlot http://vo.iucaa.ernet.in/~voi/voplot.htm VOMegaPlot

http://vo.iucaa.ernet.in/~voi/vomegaplot.htm

IUCAA http://www.iucaa.ernet.in Persistent Systems Pvt. Ltd. http://www.persistentsys.com

Sample VOTable<TABLE><FIELD name=“RAJ2000” datatype =“double” ></FIELD><FIELD name = “DEC2000” datatype=“double” ></FIELD><DATA> <TABLEDATA>

</TR><TR>

</TR><TR>

</TR></TABLEDATA></DATA></TABLE>