1
Querying Infinite Databases
Safety of Datalog Queries over infinite Databases (Sagiv and Vardi ’90)Queries and Computation on the Web (Abiteboul and Vianu ’97)
Itay Maman049011 Student Symposium, 5 July 2006
2/19
Simple Technion Queries…
(Domain: The Technion’s students database)
• Q1: Which courses did Gidi attend? SELECT course FROM students WHERE name='Gidi'
• Q2: Which students took 234218? SELECT name FROM students WHERE course='234218'
coursescourse name234218 Gidi236703 Gidi234218 Dina… …
3/19
Simple Web Queries…
• Q3: Which pages does my home page link to? SELECT target FROM links WHERE source='www.geocities.com/mysite'
• Q4: Which pages link to my home page? SELECT source FROM links WHERE target='www.geocities.com/mysite'
• Q4 is challenging: No matter how long my web-crawler works… … I can never find all incoming links of a page! This is an infinite query
• The more you crawl the more answers you get (In Q3 the size of the result set is bounded)
linksSource target www.google.com www.google.co.il www.geocities.com/mysite www.ynet.co.il www.cnn.com www.geocities.com/mysite … …
4/19
Leading questions
• What does an infinite DB look like? • Can we evaluate a query over an infinite DB?• Can we determine the finiteness of a query?
• But first, some Datalog…
5/19
Datalog• Why Datalog?
Supports recursion/transitive closure (unlike SQL)• Recursion is essential in large data-sets
Terminates if DB is finite Very simple
• program = A collection of rules• rule = A sequence of terms
• In our program: Three rules Two queries (AKA: IDB): g(X), small(X,Y) One Table (AKA: EDB): before(X,Y) A goal predicate from which execution starts
• We choose g(X) as the goal
g(X) :- small(X,2).small(X,Y) :- before(X,Y).small(X,Y) :- small(X,Z), before(Z,Y).
6/19
Finiteness
• A DB is finite If every table is a finite set before(X,Y) { (0,1), (1,2), (2,3) }
• Possible evaluation schemes: Brute force Bottom up
• Optimizations
•The Requirement: Finiteness of tables
•The guarantee: Termination of the Datalog program
7/19
Infinity
• Here is another definition for our table before(X,Y) { (X,X+1) | X 0 }
• We now have an infinite DB The Problem: we cannot iterate over the tuples in the set The solution: Top-down algorithm
• Such tables are quite common The internet links relation
links(X,Y) { (X,Y) | page X links to page Y } Java’s subclassing relation
extends(X,Y) { (X,Y) | class X extends Y }
Leading question:What does as infinite DB look like?
8/19
Example: Top-down evaluation
g(W) = s(W,2) = b(W,2) s(W,Z) b(Z,2) = {(1,2)} s(W,1) {(1,2)} = {(1,2)} [b(W,1) s(W,Z) b(Z,1)] {(1,2)} = {(1,2)} [{(0,1)} s(W,0) {(0,1)}] {(1,2)} = {(1,2)} [{(0,1)} [b(W,0) s(W,Z) b(Z,0)] {(0,1)}]
{(1,2)} = {(1,2)} [{(0,1)} [ s(W,Z) ] {(0,1)}] {(1,2)} = {(1,2)} [{(0,1)} {(0,1)}] {(1,2)} = {(1,2)} {(0,1)} {(1,2)} = {(1,2)} {(0,2)} = {(1,2),
(0,2)}
g(W) :- small(W,2).small(A,B) :- before(A,B).small(X,Y) :- small(X,Z), before(Z,Y).before(X,Y) { (X,X+1) | X 0 }
•b : before•s : small : Join
s(X,Y) = b(X,Y) s(X,Z) b(Z,Y)
9/19
Top-down evaluation• The Top-down algorithm
Init: assign r body of the goal Loop:
• (Intelligently) Pick a term, t, from r• If t is a query term:
Replace it with the union of the rules indicated by t• If t is a table term:
Replace it with the set generated by the table• Replace s expressions (in r) with • Replace s expressions (in r) with s• Evaluate relational algebra expressions (if both sides are known)
Stop if no further replacements can be made
Leading question:Can we evaluate a query over an infinite DB?
Yes
10/19
Infinite Queries• Can the top-down algorithm run forever?
Yes
• Case 1: An table that returns an infinite result evenProduct(X,Y) { (X,Y) | X*Y mod 2 = 0 } divides(X,Y) { (X,Y) | X mod Y = 0 } links(X,Y) { (X,Y) | page X links to page Y }
• weak-safety: all intermediate results are finite
• Result #1 (Sagiv and Vardi ’90): Weak-safety is decidable given F/C (finiteness constraints) of tables
• F/C of evenProduct: None• F/C of divides: X => Y• F/C of links: X => Y
Algorithm: Tracking flow of values from assigned variables
11/19
g(W) = s(2,W) = b(2,W) s(2,Z) b(Z,W) = {(2,3)} s(2,Z) b(Z,W) = {(2,3)} [b(2,Z) s(2,Z’) b(Z’,Z)] b(Z,W)…
Infinite Queries (cont.)• Can the top-down algorithm run forever?
Yes
• Case 2: The algorithm’s recursion never stops A query/table is used in its “unbounded” direction
g(W) :- small(2,W).small(A,B) :- before(A,B).small(X,Y) :- small(X,Z), before(Z,Y).before(X,Y) { (X,X+1) | X 0 }
s(X,Y) = b(X,Y) s(X,Z) b(Z,Y)
• Results #2-3 (Sagiv and Vardi ’90): Termination is undecidable in the general case Termination is decideable if all queries are unary
12/19
Infinite Queries (summ.)
• We can automatically determine weak-safety• We cannot (automatically) determine termination
• But, one can analytically prove that a given query over a given DB is finite E.g., our small(W,2) program
Leading question:Can we determine the finiteness of a query?
No
13/19
The Web as a DB
• The web data model (WDM): A scheme of a DB that can represent the web graph Just three tables:
urls = { u | u is a url of a web-page }links = { (u1,u2) | u1 links to u2; u1, u2 urls }Words = { (u,w) | w appears in page u; u urls }
• Result #4 (Abiteboul and Vianu ’97): If a Datalog program with no literals halts over
an infinite DB, its result is • => A non-trivial query (over an infinite DB) must have a literal
14/19
Web - Machines
• Browsing Machine A weakly safe Datalog program (over WDM) At least one URL literal
• Searching/Browsing Machine An unsafe Datalog program (over WDM)
• Evaluates queries in parallel Allowed literal types: URLs, Words
• Claims #1-2 (Abiteboul and Vianu ’97): Browsing machine:
• Represent a user following static links from a page Searching/Browsing machine:
• Also allows the user to access search engine
15/19
Discussion: Finite approximation• Relational Database servers are very popular
Such DBs are finite
• Also, computing a table on demand may be slow Better performance at batch processing
The challenge: Build a finite replacement for an infinite DB
• Formally: Given a finite query, q, over an infinite DB,
• (Finiteness of q proved analytically) Build a finite Database, , such that q over yield the
same result as q over
16/19
Discussion: Finite approximation
• Example: Our small(W,2) program A finite, sound table: before(X,Y) { (0,1), (1,2) } A finite, unsound table: before(X,Y) { (0,1) }
• The process: Compute the transitive closure of the before relation Start from the literal ‘2’ at the right-hand side position
• Condition: the table graph must end with a sink In before the sink is the vertex ‘0’
• => We can build a finite DB Sadly, In the web-graph no such sink exists
17/19
Discussion: Temporality• Crawling takes time• The subject may change while crawling
The DB is a snapshot which never happened
• (Open Question):• Can we decide whether a result was really “true”
at some point?
18/19
More issues
• Relational algebra over large relations BDD
• Negation Stratified Datalog
19/19
- Questions ? -
20/19
21/19
Datalog
• Semantics: ???• Straight forward mapping to Relational
Algebra??
g(X) :- small(X,2).small(X,Y) :- before(X,Y).small(X,Y) :- small(X,Z), before(Z,Y).
22/19
Example: Bottom-up evaluation
beforeX Y0 11 22 3
Initialization: Translate the EDBs into relations
23/19
Example: Bottom-up evaluation
smallX Y0 11 22 3
apply small(X,Y) :- before(X,Y).beforeX Y0 11 22 3
24/19
Example: Bottom-up evaluation
beforeZ Y0 11 22 3
apply small(X,Y) :- small(X,Z), before(Z,Y).lessX Z0 11 22 3
smallX Y0 11 22 30 21 3
Join
smallX Z0 11 22 3
beforeZ Y0 11 22 3
smallX Z0 11 22 30 21 3
smallX Z0 11 22 30 21 3
smallX Z0 11 22 30 21 30 3
25/19
Example: Bottom-up evaluation
apply g(X) :- small(X,2).smallX Y0 11 22 30 21 30 3
gX10
smallX Y0 11 22 30 21 30 3
26/19
Finitenessbefore(X,Y) { (0,1) (1,2) (2,3) }
• The Bottom-up algorithm: Init:
• For each EDB, p, assign r(p) Relation of all tuples satisfying p• For each IDB, p, assign r(p)
Loop:• Choose a rule p(…) :- t1(…), t2(…), … tn(…)• t join of all r(ti), where 1 i n• r(p) r(p) t
Continue until a fix-point is reached•Requires: Finiteness of EDBs•Ensures: Termination
