Post on 01-Feb-2016
description
transcript
VisIVO: data VisIVO: data visualization on the gridvisualization on the grid
Scientific Multidimensional Data Scientific Multidimensional Data ExplorationExploration
4th EGEE User Forum/OGF 25 and OGF Europe's 2nd International Event 2-6 March 2009Catania
Ugo Becciani, A. Costa, G. Caniglia, C. Gheller
M. Comparato, P. Massimino, M. Krokos, L. Zef Han
--fformat votable /home/user/demo/vizier.xml.....--x x --y y --z z --color --colortable --colorscalar scalar0 --glyphs sphere
VisIVODesktop
VisIVOServerVisIVOWeb
Closely integrated, complementary and independent !
Pc GNU/Linux
Grid environment
Interactive fast navigation
VisIVO Desktop VisIVO desktop is a visualisation package developed in collaboration
between INAF (Catania Astrophysical Observatory), CINECA (the largest Italian academic supercomputing center) and University of Portsmouth, with the specific object of supporting visualisation and analysis of astrophysical data. It was developed in the Italian participation in the VO-TECH project.
• Free Software
• Cross Platform: MacOSX Windows and Linux (Ubuntu , Debian, Fedora),
• N-Dim (>3) Visualization of Astrophysical data (lut and glyphs =>
scalar properties of each element)
• Point, Volume and Vector Visualization
• Astrophysical Data Analysis
• VO (Virtual Observatory) compliance (Interoperability: VOT,
PLASTIC,VizieR)
• Easy to use: intuitive menu drives the user.
➲ Input file formats
Ascii, CSV Binary HDF5 FITS Gadget VOTable
Uniform management
All internal data are represented like a table, a sequence of float values (VisIVO Binary Table - VBT)
VisIVO Desktop
Visualisations Navigation -- Zoom -- Lookup table -- Algorithms -- Data selection -- Picker op. --
Interoperability
LODLevel Of Detail: while interacting with the visualized data (i.e.
zooming and rotating) a simplified graphical representation is used in order to keep the manipulation interactive
Data manipulation New Fields can be created as
mathematical functions of existing fields
Tables can be merged
Subsets can be extracted using different sampling techniques
Vectors
Vector Fields can be created from a VBT selecting any loaded fields as its three components
VisIVO Server on the Grid Non-interactive command line application that implements
visualisation functionalities; its output is a static 2D image of a 3D object
Easy integration in Virtual Observatory complaint Web Services and in the Grid environment
It will provide the user with a 3D preview of huge data exploiting the powerful facilities of the Grid environment
Easy to use
Open Source code: project maintained on sourceforge. NO LIMIT on data size !
VisIVO ServerBasic Architecture
TVO XML Document
Local or Remote (http ref) User Data
VBT: VisIVO Binary TableVBT: VisIVO Binary Table
New VBTNew VBT
VisIVO ImporterInput: Local or Remote file: ASCII-CSV, Binary, VOTable, FLY, Gadget, TVO XML Document , Fits Table, HDF5
Op: Data format conversion: VBTs creation
Output: VBTs (optimised for VisIVO Filters and Viewer).
NOTE: No limit on data dimension for input data
float616777216littleXYZVx...
float616777216littleXYZVx...
VBT: VisIVO Binary Table Ascii Header (.bin.head):contains basic metadata information.
Internal data rappresentation: floatNumber of fieldsNumber of elements of each field (including mesh data for volumes)Data format identifier: little endianList of field names
VisIVO Server
Binary File (.bin):a sequence of binary arrays (fields).
VisIVO Importer VisIVOImporter --fformat filetype [--out name] [--volume] [--compx X --compy Y --compz Z] [--sizex DX --sizey DY --sizez DZ] [--bigendian] [--double] [--npoints N] filename
--fformat specifies the input file data format--out [name] (optional) output filename. Default name VisIVOServerBinary.bin--volume (optional) used to create a table that contains Volume data--compx [double] --compy [double] --compz [double] (optional) If data is a Volume the user must enter the mesh geometry. If data size fits a cubic mesh these valuesare automatically computed.
--sizex [double] --sizey [double] --sizez [double] (optional). Cell geometry Default values 1.0 1.0 1.0
--bigendian (optional) Used only for big Endian Gadget and FLY input file. Default files are little Endian.
--double (optional) Used only for double data of FLY input file. Default value is float--npoints (optional) Used only for FLY input file. Number of data points.
VisIVO FiltersInput: A VBT data file
Op: It creates a new VBT or gives info on specified fields
Output: a new VBT or other info
NOTE: it must be applied to visualise huge data
General Syntax:
VisIVOFilters --op filterOpCode <options> InputFile (VBT file)
VisIVO FiltersAppend Tables: Creates a new table appending data from a list of existing tables Coarse Volume: Produces a coarsed subvolume with plane extraction from the original volumeDecimator: Creates a subtable as a regular subsample from the input table.Extract Subregion: Creates a new table from an input table: sub-box or sphere extraction.Extract Subvolume: Produces a table which represents a subvolume from the original volumeMath. Operations: Create a new field in a data table as the result of a mathematical operation between existing fieldsMerge Tables: Create a new table from two or more existing data tablesPoint Distribution: Creates a volume using a field distribution (CIC algorithm) on a regular mesh
Point Property: Assigns a property using a field distributionRandomizer: Creates a random subset from the original data tableSelect Fields: Creates a new table using one or more fields of a data table Select Rows: Creates a new table using limits on one or more fields of a data table Show Table: Produces an ASCII table of selected fields Statistic: Creates and plots an histogram of a scalar field in the table
VisIVO ViewerVisIVO Viewer is a command line application that produces
3D images from the binary internal data format table (VBT)
The user must specify three fields of the table for 3D representation.
The user can also specifiy the following main options:- Camera position (azimuth and elevation)- Zoom level- Transparency / Opacity- Point shape (pixel/sphere, cube, cone etc..)- Lookup table- ..........
VisIVO ViewerGeneral Syntax
VisIVOViewer <options> inputFile (VBT file)
--volume (for volume representation)
Input: A VBT data file
Op: It creates one or more view of data
Output: png files
LIMITS: the input VBT data file must fit the available RAM
- Default Output: 4 Images:
- Az=0 - Elev=0- Az=90 - Elev=0- Az=0 - Elev=90- Az=45 -Elev=45
VisIVO Viewer
Customized Image fixing: Az, Elev, Transparency....
Cometa Consortium
COnsorzio Multi Ente per la promozione e
l'adozione di Tecnologie di calcolo Avanzato
PI2S2 Project
Funded by the Italian MUR using EU funds for Objective 1
The Sicilian Grid in one slide
Available computing and storage resources 1. ~2500 cores AMD Opteron 22182. 2 GB RAM per core 3. LSF as LRMS everywhere4. Infiniband-4X at all sites
1. 200+ TB of raw disk storage 2. Distributed/parallel GPFS filesystem
gLite 3.0 middleware everywhere
PI2S2 infrastructure and HPC
• Each PI2S2 site has a low latency Infiniband network
• Intel fortran and PGI compilers are available
UI “on-board”: share all resources with the WNs available remote login it is possible to compile and test codes
HPC RUN
A special hpc queue is available. It allows the CPU core reservation
It is possible to use ALL nodes of a PI2S2 site (more than 200 CPU cores)
• PI2S2 Grid infrastructure supports the following MPI versions
PI2S2 infrastructure and HPC
each MPI version has been built with different compilers. The supported tags are :
MVAPICH_GCC4 MVAPICH_INTEL9 MVAPICH_PGI706
MVAPICH2_GCC4 MVAPICH2_INTEL9 MVAPICH2_PGI706
MPICH_GCC4 MPICH_INTEL9 MPICH_PGI706
o MPICH_SH_GCC4 o MPICH_SH_INTEL9 o MPICH_SH_PGI706
MPICH2_GCC4 MPICH2_INTEL9 MPICH2_PGI706
Running Using Gigabit Network
Running Using Inifiband 4x Network
Type = "Job";
JobType = "MVAPICH2";
Executable = “MPIparallel_exec”;
NodeNumber = 200;
Arguments = “arg1 arg2 arg3";
StdOutput = "test.out";
StdError = "test.err";
InputSandbox = {“mpi.pre.sh”,“mpi.post.sh”,
“MPIparallel_exec”, “watchdog.sh”};
OutputSandbox = {“test.err”, “test.out” }
PI2S2 infrastructure and HPC
Pre e Post Processing Scripts
Pre e Post Processing Scripts
ExecutableExecutable
It runs during the parallel executionIt runs during the parallel execution
VisIVOServer
HPC and Visualisation on the Grid
- Can be executed on the Grid
- Produces images (on the grid Catalog)
- Visualisation of large dataset during the data production phase
Data Image, obtained with VisIVO Server, of a cosmological half-billion bodies simulation, running since January 9th on the Cometa Grid
Independent job
Sub-process of the watchdog procedure
VisIVOWeb
Ugo BeccianiPrincipal InvestigatorAstrophysical Observatory Catania Italy
Mel KrokosCo-Principal InvestigatorUniversity of Portsmouth UK
Claudio Gheller Co-Principal InvestigatorCineca Bologna Italy
Gabriella Caniglia Software Developer Astrophysical Observatory Catania Italy
Marco Comparato Software Manager/Developer Astrophysical Observatory Catania Italy
Alessandro Costa Web Services & TVO Astrophysical Observatory Catania Italy
Zef Han J.Software developerUniversity of Portsmouth UK
Pietro MassiminoWeb developerAstrophysical Observatory Catania Italy
http://visivo.oact.inaf.it
http://visivoserver.oact.inaf.it
Thanks!