Post on 13-Feb-2016
description
transcript
Ohio State University Department of Computer Science and Engineering
Automatic Data Virtualization - Automatic Data Virtualization - Supporting XML based abstractions Supporting XML based abstractions
on HDF5 Datasetson HDF5 DatasetsSwarup Kumar Sahoo
Gagan Agrawal
Ohio State University Department of Computer Science and Engineering
RoadmapRoadmap• Motivation • Introduction• System Overview• XQuery, Low and High Level schema and HDF5
storage• Compiler Analysis and Algorithm• Experiment • Summary and Future Work
Ohio State University Department of Computer Science and Engineering
MotivationMotivation• Emergence of grid-based data repositories
– Can enable sharing of data• Emergence of applications that process large datasets
– Complicated by complex and specialized storage formats• Need for easily portable applications
– Compatibility with web/grid services
Ohio State University Department of Computer Science and Engineering
Data VirtualizationData Virtualization An abstract view of data
datasetData Service
DataVirtualization
By Global Grid Forum’s DAIS working group:• A Data Virtualization describes an abstract view of data.• A Data Service implements the mechanism to access and process data through the Data Virtualization
Ohio State University Department of Computer Science and Engineering
Introduction : Automatic Data Introduction : Automatic Data VirtualizationVirtualization
• Goal : Enable Automatic creation of efficient data services
– Support a high-level or abstract view of data– Data is stored in low-level format
• Application development: – assume a high-level or virtual view
• Application Execution: – On actual low-level layout
Ohio State University Department of Computer Science and Engineering
Overview of Our Automatic Data Overview of Our Automatic Data Virtualization WorkVirtualization Work
• Previous work on XML Based virtualization – Techniques for XQuery Compilation (Li and Agrawal, ICS
2003, DBPL 2003) – Supporting XML Based high-level abstraction on flat-file
datasets (LCPC 2003, XIME-P 2004)• Relational Table/SQL Based Implementation
– Supporting SQL Select and Where (HPDC 2004) – Supporting SQL-3 Aggregations (LCPC 2004)
Ohio State University Department of Computer Science and Engineering
XML-based VirtualizationXML-based Virtualization
TEXT
…
NetCDF
RDBMS
HDF5
XML
XQuery
???
Ohio State University Department of Computer Science and Engineering
Challenges and ContributionsChallenges and Contributions
• Challenges – Compiler generates efficient data processing code
» Uses the information about the low-level layout and mapping between virtual and low-level layout
– Challenge in compilation» High level to low level » to ensure high locality in processing of large datasets
• Contributions of this paper – An improved data- centric transformation algorithm – An implementation specific to HDF5 as the low-level format
Ohio State University Department of Computer Science and Engineering
System OverviewSystem OverviewHigh levelXML Schema
Mapping Schema
XQuery Source Code
Compiler
Generated Code
Processor and Disk
System OverviewSystem OverviewLow levelXML Schema
HDF5 Library
Ohio State University Department of Computer Science and Engineering
XQuery and HDF5XQuery and HDF5
• High-level declarative languages ease application development– XQuery is a high-level language for processing XML datasets– Derived from database, declarative, and functional languages!
• HDF5:– Hierarchical Data Format – Widely used in scientific communities – A case study with a format which has optimized access
libraries
Ohio State University Department of Computer Science and Engineering
Use of XML SchemasUse of XML Schemas
• High-level schema– XML is used to provide a virtual view of the dataset
• Low-level schema – reflects actual physical layout in HDF5
• Mapping schema:– describes mapping between each element of high-level
schema and low-level schema
Ohio State University Department of Computer Science and Engineering
Oil Reservoir SimulationOil Reservoir Simulation• Support cost-effective Oil
Production• Simulations on a 3-D grid• 17 variables and cell
locations in 3-D grid at each time step
• Computation of bypassed regions
– Expression to determine if a cell is bypassed for a time-step
– Within a spatial region and range of time steps
– Grid cells that are bypassed for every time-step in the range
Oil Reservoir management
Ohio State University Department of Computer Science and Engineering
High-Level SchemaHigh-Level Schema< xs:element name="data" maxOccurs="unbounded" >
< xs:complexType > < xs:sequence >
< xs:element name="x" type="xs:integer"/ > < xs:element name="y" type="xs:integer"/ > < xs:element name="z" type="xs:integer"/ > < xs:element name="time" type="xs:integer"/ > < xs:element name="velocity" type="xs:float"/ > < xs:element name="mom" type="xs:float"/ >
< /xs:sequence > < /xs:complexType >
< /xs:element >
Ohio State University Department of Computer Science and Engineering
High-Level XQuery Code Of Oil High-Level XQuery Code Of Oil Reservoir managementReservoir management
unordered( for $i in ($x1 to $x2)
for $j in ($y1 to $y2) for $k in ($z1 to $z2)
let $p := document("OilRes.xml")/datawhere ($p/x=$i) and ($p/y = $j) and ($p/z = $k) and ($p/time >= $tmin) and ($p/time <= $tmax) return <info> <coord> {$i, $j, $k} </x-coord> <summary> { analyze($p) } </summary> </info>
)
Ohio State University Department of Computer Science and Engineering
Low-Level SchemaLow-Level Schema<file name="info">
<sequence> <group name="data">
<attribute name="time"> <datatype> integer </datatype> <dataspace> <rank> 1 </rank> <dimension> [1] </dimension> </dataspace> </attribute>
<dataset name="velocity"> <datatype> float </datatype> <dataspace> <rank> 1 </rank> <dimension> [x] </dimension> </dataspace> </dataset>
..............
</group> </sequence>
</file>
Ohio State University Department of Computer Science and Engineering
Mapping SchemaMapping Schema//high/data/velocity //low/info/data/velocity //high/data/time //low/info/data/time //high/data/mom //low/info/data/mom [index(//low/info/data/velocity, 1)]
//high/data/x //low/coord/x [index(//low/info/data/velocity, 1)]
Ohio State University Department of Computer Science and Engineering
Compiler AnalysisCompiler Analysis
• Problem with direct translation :– Each let expression involves complete scan over dataset– So final code will need several passes over the data
• Solution :– Apply Data Centric Transformations to read a portion HDF5
dataset only once
Ohio State University Department of Computer Science and Engineering
Naïve Strategy Naïve Strategy
DatasetOutput
Requires 3 Scans
Ohio State University Department of Computer Science and Engineering
Data Centric StrategyData Centric Strategy
DatasetsOutput
Requires just one scan
Ohio State University Department of Computer Science and Engineering
Data Centric TransformationData Centric Transformation
• Overall Idea in Data-Centric Transformation – Iterate over each data element in actual storage – Find out iterations of the original loop in which they are accessed.– Execute computation corresponding to those iterations.
• Previous Work – Pingali et al.: blocking – Ferreira and Agrawal: data-parallel Java on disk-resident datasets– Li and Agrawal: XQuery, invert getData functions
• Our contribution: – Use Low-Level and Mapping Schema – Extend the idea when multiple datasets need to be accessed
Ohio State University Department of Computer Science and Engineering
Data Centric TransformationData Centric Transformation
• Mapping Function T :Iteration space → High-Level data
• Mapping Function C : High-Level data → Low-Level data
• Mapping Function C · T = M : Iteration space → Low-Level data
• Our Goal is to compute M-1.
Ohio State University Department of Computer Science and Engineering
Data Centric TransformationData Centric Transformation
• Choose one dataset as base dataset S1 from n datasets to be accessed
• Apply M1-1 to compute set of iterations.
• The expression Mi · M1
-1 gives the portion of dataset Si that needs to be accessed along with S1
• Choice of base dataset might impact the data locality.
Ohio State University Department of Computer Science and Engineering
Choice of Base DatasetChoice of Base Dataset
• Min-IO-Volume Strategy – Minimize repeated access to any dataset
• Min-Seek-Time Strategy – Minimize any discontinuity in access
Ohio State University Department of Computer Science and Engineering
Template for Generated CodeTemplate for Generated CodeGenerated_Query { Create an abstract iteration space using Source code. Allocate and initialize an array of output element corresponding to
iteration space. For k = 1, …, NO_OF_CHUNKS
{ Read kth chunk of dataset S1 using HDF5 functions and structural tree. Foreach of the other datasets S2, … , Sn
access the required chunk of the dataset. Foreach data element in the chunks of data
{ compute the iteration instance. apply the reduction computation and update the output.
} }
}
Ohio State University Department of Computer Science and Engineering
ExperimentExperimentImpact of Strategy and Chunk-Size, Dataset1
0200400600800
100012001400
1 5 15 31 62 125
Read Chunk-Size(x1000 elements)
Tim
e(se
c)
)
Min-Seek-Time
Min-IO-Volume
200*200*200 grid with 10 time steps (1.28 GB)
50*50*50 Storage Chunk Size
Ohio State University Department of Computer Science and Engineering
ExperimentExperimentImpact of Strategy and Chunk-Size, Dataset2
0
50
100
150
200
250
300
350
1 5 15 31 62
Read Chunk-Size(x1000 elements)
Time(
sec)
)
Min-Seek-TimeMin-IO-Volume
50*50*50 grid with 200 time steps (400 MB)
25*25*25 Storage Chunk Size
Ohio State University Department of Computer Science and Engineering
Key ObservationsKey Observations
• Overall minimum execution time – Min-IO-Volume strategy when read chuck size matches
storage chunk size
• Execution time – Very sensitive to Read Chunk-Size in Min-IO-Volume
Strategy– Not sensitive to Read Chunk-Size in Min-Seek-Time
Strategy due to buffering of Storage chunks
Ohio State University Department of Computer Science and Engineering
SummarySummary• Compiler techniques
– Support High-level abstractions on complex low-level data formats
– Enables use of the same source code across a variety of data formats
– Perform data centric transformations automatically– Experimental result shows minor change in strategy can affect
performance significantly • Future Work
– Cost models to guide strategy and chunk size selection – Compare performance with manual implementations – parallelizing data processing– extend applicability of the algorithm to more general class of
queries