Ten Thousand SQLs

transcript

Ten Thousand SQLs

Kalmesh Nyamagoudar2010MCS3494

2 October 13, 2011

• Example• Definitions• Algorithm

CN Generatio

n• Sequential

Algorithm• CLP : Naïve• CLP : New• OLP• DLP• Performance

Studies

CN Evaluation

CONTENTS

3 October 13, 2011

BANKS Model

Author1

Author2

Paper1

Author1

Author2

Paper2Steiner Trees

4 October 13, 2011

DISCOVER Model

Author1 Author2

Paper1

AUTHOR WRITES PAPER CITE

Writes

Paper{}

Writes

Joining Network Of Tuples

Joining Network Of Tuple Sets

Author1: Paper1 Author2: Paper1

Author1 Author2

Paper2

Author1: Paper2 Author2: Paper2

AuthorAuthor1AuthorAuthor2

AuthorAuthor1 Writes

Paper{}

⋈Writes

⋈AuthorAuthor2

Database n Relations Each has attributes

Schema Graph : Directed graph that captures p-f relationships in

database schema : Each relation : for each p-f relationship Assumption : No self loops/parallel edges Undirected version of (Future reference : )

Background : DISCOVER

October 13, 2011

Schema

(TPC-H)

October 13, 2011

Background : DISCOVERExample Data

ORDERS ORDERKEY CUSTKEY TOTALPRICE CLERK ... 1000105 12312 $5,000 John Smith 1000111 12312 $3,000 Mike Miller 1000125 10001 $7,000 Mike Miller 1000110 10002 $8,000 Keith Brown

CUSTOMER CUSTKEY NAME NATIONKEY ...

12312 Brad Lou 01

10001 George Walters 01

10013 John Roberts 01

NATION NATIONKEY NAME REGIONKEY 01 USA N.America

Source : Discover[3]

October 13, 2011

Background : DISCOVER Query: Smith,Miller”

12312 Brad Lou 01

October 13, 2011

12312 Brad Lou 01

RESULT

2 O1 C1 O2

October 13, 2011

12312 Brad Lou 01

SIZE RESULT

2 O1 C1 O2

4 O1 C1 N1 C2 O3

Joining Network Of Tuples

October 13, 2011

11 October 5, 2011

Background : DISCOVERSource : Discover[2]

Final Answer : Joining Network Of Tuples () Tree of tuples For each pair of adjacent tuples , where , , there is an edge in

schema graph and (

Keyword Query Given : Set of keywords Result : Set of all possible joining networks of tuples that are both :

Total : every keyword is contained in at least one tuple of joining network.

Minimal : Removal of any tuple still gives a TJNT? Ordered By : Size of MTJNTs

All such joining network of tuples are Minimal Total Joining Network Of Tuples (MTJNT)

October 13, 2011

Tree of tuple sets

For each pair of adjacent tuple sets , there is an edge in schema

Candidate Network

Given : Set of keywords

Is a Joining Network Of Tuple Sets such that there is an instance I

of the database that has a MTJNT and no tuple that maps to a

free tuple set contains any keywords

2 Steps :

candidate networks (CNs) generation

CNs evaluation

October 13, 2011

Candidate Networks Generation Complete : Every possible MTJNT is produced by

a candidate network output by the algorithm Minimal : Does not produce any redundant

candidate networksExample:

ORDERSSmith ⋈ CUSTOMER{} ⋈ ORDERSMiller

ORDERSSmith ⋈ CUSTOMER{} ⋈ ORDERSMiller ⋈ CUSTOMER{} ORDERSSmith ⋈ CUSTOMER{} ⋈ ORDERS{} ORDERSSmith ⋈ LINEITEM{} ⋈ ORDERSMiller

Tmax : Maximum number of tuple sets in a CN

October 13, 2011

CN Generation

October 13, 2011

CN Generation

October 13, 2011

CN Generation

October 13, 2011

Large number of CNs to be evaluated CNs : usually tightly coupled with each other Reuse the common sub expressionsExample :

Dataset : DBLP No. of tables : 4 Max No. of tuples/result : 11 CN join operations without sharing : 539,052 CN join operations with sharing : 53,008 probability for any two CNs to share computational

cost : 59.64

CN Evaluation : October 13, 2011

Sequential Algorithm :Example

Dataset : DBLP

Source : TTS[1]

AUTHOR WRITE

PAPER CITE

October 13, 2011

Keywords Entered : CNs generated :

Source : TTS[1]

Sequential Algorithm :Example

AUTHOR WRITE PAPER CITE

October 13, 2011

CN Evaluation : state-of-art sequential algorithm

Greedy algorithm:

In each iteration build intermediate result of size that

maximizes

No of occurrences of IMR in CNs

Estimated no. of tuples of IMR

gives better results

October 13, 2011

Example CN :

Total Cost : 2199

Source : TTS[1]

Sequential Algorithm :Execution Graph

October 13, 2011

: DAG ∈ V(GE) : An operator(e.g. or σ) ∈ E(GE)) iff output of is part of the input of

Levels σ : level 1 A node is in level iff

∃, such that ∈ E(GE) and is in level , and

∀, such that ∈ E(GE), the level of is no larger than

: Maximum level of GE Evaluation

Evaluated in a bottom-up fashion No parallelism involved For keyword queries with large number of keywords or with high selective

keywords, the processing time is slow

Sequential Algorithm :Execution Graph

October 13, 2011

New Solution

Use of multi-core architecture

Why not existing parallel multi-query processing? Large number of queries Large sharing between queries Large intermediate results

What we need on multi-core archs? CNs in the same core share : most computational cost CNs in different cores share : least computational cost Handle high workload skew Handle errors caused by estimation adaptively

October 13, 2011

CN Level Parallelism :Straightforward Approach largest first rule : partition with the least workload

Final Cost : max(cost of each core) = 1949

Source : TTS[1]

October 13, 2011

CLP : Straightforward Approach

Execution Time =

Problem : does not consider sub-expression sharing

Source : TTS[1]

select the core : O(n) Add CN to partition :O()

October 13, 2011

CLP:Sharing-Aware CN Partitioning

Which CN to distribute first? the largest not-shared/extra cost

To which partition? with maximum sharing if it does not destroy the workload

balancing.

Total cost for a partition = cost after sharing sub-expressions for all CNs in that partition

October 13, 2011

A P P A P P

𝜎 𝑘1 W 𝜎 𝑘1 C 𝜎 𝑘1 C 𝜎 𝑘2 W 𝜎 𝑘2 C 𝜎 𝑘2 C

⋈ ⋈ ⋈ ⋈ ⋈ ⋈

⋈ ⋈

⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈𝑐1 𝑐2 𝑐3 𝑐4 𝑐5 𝑐6 𝑐7 𝑐8 𝑐9

Core 1 Core 2 Core 3

CN MinCost

10 10 10 100

MaxHeap5 5 5 5 5 5 5 5 5

: Non-Exec Graph of Core 3October 13, 201128

A P P A P P

𝜎 𝑘1 W 𝜎 𝑘1 C 𝜎 𝑘1 C 𝜎 𝑘2 W 𝜎 𝑘2 C 𝜎 𝑘2 C

⋈ ⋈ ⋈ ⋈ ⋈ ⋈

⋈ ⋈

⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈𝑐1 𝑐2 𝑐3 𝑐4 𝑐5 𝑐6 𝑐7 𝑐8 𝑐9

10 10 10 100

MaxHeap

(727) (727) (715)

CN MinCost

5 5 5 5 5 5 5 5 5

October 13, 201129

A P P P

𝜎 𝑘1 W 𝜎 𝑘1 C W 𝜎 𝑘2 C 𝜎 𝑘2 C

⋈ ⋈ ⋈ ⋈

⋈ ⋈

⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈𝑐1 𝑐2 𝑐3 𝑐4 𝑐5 𝑐6 𝑐7 𝑐8 𝑐9

(727) (727) (715)

CN MinCost

10 10 102

5 5 5 5 5 5 5 5 5 MaxHeap

October 13, 201130

𝜎 𝑘1 C W 𝜎 𝑘2 C 𝜎 𝑘2 C

⋈ ⋈ ⋈

⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈𝑐1 𝑐2 𝑐3 𝑐4 𝑐5 𝑐6 𝑐7 𝑐8 𝑐9

CN MinCost

10 102

5 5 5 5 5 5 5 5 5 MaxHeap

(727) (727) (715)

(842) (835)

(1442) (1447) (950)

October 13, 201131

W 𝜎 𝑘2 C 𝜎 𝑘2 C

⋈ ⋈

⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈ ⋈𝑐1 𝑐2 𝑐3 𝑐4 𝑐5 𝑐6 𝑐7 𝑐8 𝑐9

CN MinCost

5 5 5 5 5 5 5 5 5

(727) (727) (715)

(842) (835)

MaxHeap

October 13, 201132

Total Cost = 957

43.5 of the sequential cost

Source : TTS[1]

October 13, 2011

Execution Time = [ Assuming ]

Redundant works are done by multiple cores

Wrong Estimation : Accumulated Cost

Source : TTS[1]Initialization

Select the core :

Update cost of overlapping CNs :|E(GE)|n overall

October 13, 2011

CLP:Error Accumulation

Source : TTS[1]

October 13, 2011

Operator Level Parallelism

Each CN is allowed to be processed in different cores, but each

operation must be processed in a certain core

Nodes in the same phase processed parallely

Time Of Partition?

In phase if node is in level of GE

Which operation to distribute?

the largest cost

To which partition?

Minimum cost if it does not destroy the workload balancing

Sharing between CNs and phases : Shared Memory

October 13, 2011

Final Cost = 737

33.5 of the sequential cost

Source : TTS[1]

October 13, 2011

OLP : Overcoming Error Accumulation

Before each phase, re-estimate the cost of each operation

Cost of a select operation

Cost of a join operation can be pre-computed and saved beforehand for each edge

October 13, 2011

OLP : Overcoming Accumulated Cost

Source : TTS[1]

October 13, 2011

Execution Time

A join operation is much more costly than others => becomes the dominant cost when processing

Source : TTS[1] nodes overall

October 13, 2011

Data Level Parallelism

each operation in GE can be performed on multiple cores

uses the operation level parallelism if there is no workload skew

partition data adaptively before each time workload skew

happens

Which node to partition?

Most costly node if its dominant

When to merge the sub-results?

At final phase

October 13, 2011

Source : TTS[1]Core 1Core 2Core 3

October 13, 2011

Lemma : In each phase, at most partition operations will be performed Each node will have max copies <= Execution Time = with =

Source : TTS[1]

Divide the tuples of child node

Select the child node to be partitioned

Makes copies of selected child node and all its father nodesAdds corresponding edges

Re-estimate

October 13, 2011

Performance Studies

For LINEAR processing

processing time for the state of art sequential algorithm no of cores

Implemented In

System Configuration

October 13, 2011

Default values(IMDB) : 3 ranges from 2 to 6 with a default value 4 ranges from 4 to 7 with a default value 5

Source : TTS[1]

Performance Studies

October 13, 2011

Vary (IMDB)

Source : TTS[1]

October 13, 2011

Vary (IMDB)

Source : TTS[1]

October 13, 2011

Vary (IMDB)

Source : TTS[1]

October 13, 2011

References1. Lu Qin, Jeffrey Xu Yu, Lijun Chang, Ten Thousand

SQLs: Parallel Keyword Queries Computing, Proceedings of the VLDB Endowment, Volume 3 Issue 1-2, September 2010 , Singapore

2. Vagelis Hristidis, Yannis Papakonstantinou, Discover: keyword search in relational databases, VLDB '02 Proceedings of the 28th international conference on Very Large Data Bases, Hong Kong

3. [PPT] DISCOVER: Keyword Search in Relational Databases

October 13, 2011

THANK YOU

October 13, 2011

Ten Thousand SQLs

Documents