188
Ghislain Fourny Big Data Fall 2018 12. Querying pinkyone / 123RF Stock Photo
Ghislain Fourny
Big Data Fall 201812. Querying
pinkyone / 123RF Stock Photo
2
Declarative Languages
What vs. How
3
Functional Languages
Expression
for if thenelse where
order by
whileany
every
let return
exit with
=
+
4
Ever played Lego?
5
Ever played Lego?
if( )
then
else
6
Ever played Lego?
if( )
then
else
my:func( )
7
Ever played Lego?
if( )
then
else
my:func( )
8
Ever played Lego?
if( )
then
else
my:func( )2
9
Ever played Lego?
if( )
then
else
for $x in
let $y :=
return
my:func( )2
10
Ever played Lego?
if( )
then
else
for $x in
let $y :=
return
my:func( )2
11
Ever played Lego?
if( )
then
else
for $x in
let $y :=
return
my:func( )2
12
Language ecosystemXML JSON
Navigation XPath JSONPathJSONSelect
13
Language ecosystemXML JSON
Navigation XPath JSONPathJSONSelect
Transform XSLT JSONT
14
Language ecosystemXML JSON
Navigation XPath JSONPathJSONSelect
Transform XSLT JSONTQuery XQuery 1.0/3.0 XQuery 3.1,
JSON Query, JSONiq
15
Language ecosystemXML JSON
Navigation XPath JSONPathJSONSelect
Transform XSLT JSONTQuery XQuery 1.0/3.0 XQuery 3.1,
JSON Query, JSONiq
Update, Scripting
XQuery Update Facility & Scripting
JSONiq
16
Try it out!
17
XML Navigation (XPath, XQuery)
18
Document example<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
doc("myfile.xml")
19
Document example<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
Remember:This is a tree!
doc("myfile.xml")
20
The slash operator<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
doc("myfile.xml")/countries
21
The slash operator<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
doc("myfile.xml")/countries/country
22
The slash operator<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
doc("myfile.xml")/countries/country/name
23
Descendant axis<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
doc("myfile.xml")//country/name
24
Descendant axis<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
doc("myfile.xml")//name
25
The slash operator<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
doc("myfile.xml")//country
26
Attribute axis<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
doc("myfile.xml")//country/@code
27
Explicit atomization<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
doc("myfile.xml")//country/data(@code)
28
Atomization<name>Switzerland</name>0<name>France</name>0<name>Germany</name>0<name>Italy</name>0<name>Austria</name>00
Switzerland,France,Germany,Italy,Austria,,,
data(…)
29
Explicit atomization<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
doc("myfile.xml")//country/data(@code)
30
Filter expressions<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
doc("myfile.xml")//country/@code[data(.) = "CH"]
31
Parent abbreviation<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
doc("myfile.xml")//country/@code[data(.) = "CH"]/..
32
Alternate possibility<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
doc("myfile.xml")//country[@code/data(.) = "CH"]
33
Explicit element atomization<?xml&version="1.0"&encoding="UTF98"?>&<countries>&&&<country&code="CH">&&&&&<name>Switzerland</name>&&&</country>&&&<country&code="F">&&&&&<name>France</name>&&&</country>&&&<country&code="D">&&&&&<name>Germany</name>&&&</country>&&&<country&code="I">&&&&&<name>Italy</name>&&&</country>&&&<country&code="A">&&&&&<name>Austria</name>&&&</country>&</countries>&&
doc("myfile.xml")//country[@code/data(.) = "CH"]/name/data(.)
34
With collections of trees... <?xml version="1.0" encoding="UTF-8"?> <country code="CH"> <name>Switzerland</name> </country> <?xml version="1.0" encoding="UTF-8"?> <country code="F"> <name>France</name> </country> <?xml version="1.0" encoding="UTF-8"?> <country code="D"> <name>Germany</name> </country> <?xml version="1.0" encoding="UTF-8"?> <country code="I"> <name>Italy</name> </country> <?xml version="1.0" encoding="UTF-8"?> <country code="A"> <name>Austria</name> </country>
collection("countries.xml")
35
With collections of trees... <?xml version="1.0" encoding="UTF-8"?> <country code="CH"> <name>Switzerland</name> </country> <?xml version="1.0" encoding="UTF-8"?> <country code="F"> <name>France</name> </country> <?xml version="1.0" encoding="UTF-8"?> <country code="D"> <name>Germany</name> </country> <?xml version="1.0" encoding="UTF-8"?> <country code="I"> <name>Italy</name> </country> <?xml version="1.0" encoding="UTF-8"?> <country code="A"> <name>Austria</name> </country>
collection("countries.xml")/country/name
36
JSON Navigation (JSONiq, Sparksoniq)
37
With a JSON document
{"countries" :
[{
"name" : "Switzerland","code" : "CH"
},{
"name" : "France","code" : "F"
},{
"name" : "Germany","code" : "D"
},{
"name" : "Italy","code" : "I"
},{
"name" : "Austria","code" : "A"
}]
}parse-json(file:read-text("myfile.json"))
38
With a JSON document
parse-json(file:read-text("myfile.json")).countries
{"countries" :
[{
"name" : "Switzerland","code" : "CH"
},{
"name" : "France","code" : "F"
},{
"name" : "Germany","code" : "D"
},{
"name" : "Italy","code" : "I"
},{
"name" : "Austria","code" : "A"
}]
}
39
With a JSON document
parse-json(file:read-text("myfile.json")).countries[]
{"countries" :
[{
"name" : "Switzerland","code" : "CH"
},{
"name" : "France","code" : "F"
},{
"name" : "Germany","code" : "D"
},{
"name" : "Italy","code" : "I"
},{
"name" : "Austria","code" : "A"
}]
}
40
With a JSON document
{"countries" :
[{
"name" : "Switzerland","code" : "CH"
},{
"name" : "France","code" : "F"
},{
"name" : "Germany","code" : "D"
},{
"name" : "Italy","code" : "I"
},{
"name" : "Austria","code" : "A"
}]
}parse-json(file:read-text("myfile.json")).countries[].name
41
With a JSON document
{"countries" :
[{
"name" : "Switzerland","code" : "CH"
},{
"name" : "France","code" : "F"
},{
"name" : "Germany","code" : "D"
},{
"name" : "Italy","code" : "I"
},{
"name" : "Austria","code" : "A"
}]
}parse-json(file:read-text("myfile.json")).countries[].code
42
With a JSON document
{"countries" :
[{
"name" : "Switzerland","code" : "CH"
},{
"name" : "France","code" : "F"
},{
"name" : "Germany","code" : "D"
},{
"name" : "Italy","code" : "I"
},{
"name" : "Austria","code" : "A"
}]
}parse-json(file:read-text("myfile.json")).countries[][$$.code = "CH"]
43
With a JSON document
{"countries" :
[{
"name" : "Switzerland","code" : "CH"
},{
"name" : "France","code" : "F"
},{
"name" : "Germany","code" : "D"
},{
"name" : "Italy","code" : "I"
},{
"name" : "Austria","code" : "A"
}]
}parse-json(file:read-text("myfile.json")).countries[][$$.code = "CH"].name
44
With a JSON collection {"name" : "Switzerland","code" : "CH"
}{
"name" : "France","code" : "F"
}{
"name" : "Germany","code" : "D"
}{
"name" : "Italy","code" : "I"
}{
"name" : "Austria","code" : "A"
}
collection("countries")[$$.code = "CH"].name
45
With a JSON Lines file
{ "name" : "Switzerland", "code" : "CH" }{ "name" : "France", "code" : "F" }{ "name" : "Germany", "code" : "D" }{ "name" : "Italy", "code" : "I" }{ "name" : "Austria", "code" : "A" }
json-file("countries.json")[$$.code = "CH"].name
46
Construction(XPath, XQuery)
47
Construction: Strings
"foo"
48
Construction: String Escaping (with JSONiq)
"This is a line\nand this is a new line"
"This is a \"quote\""
49
Construction: String Escaping (with XQuery)
"This is a line
and this is a new line"
"This is a "quote""
50
Construction: Numbers
42
3.141592653589793238462650283279
-6.022E23
51
Construction: Booleans
true()
false()
true
false
XQuery JSONiq
52
Construction: Other Simple Types
date("2013-05-01")
dateTime("2013-06-21T05:00:00Z")
long("1234567890123")
53
Construction: JSON (JSONiq)
[{ "foo" : "bar" },{ "bar" : [ 1, 2, true, null ]
]
54
Construction: XML (XQuery)
<foo attr="value">Text<bar/><!-- comment --><?target pi?>
</foo>
55
Construction: sequences (JSONiq)
[ 2, 3 ], true, "foo", { "f" : 1 }
56
Construction: sequences (XQuery)
1, true(), "foo", <bar/>
57
Construction: sequences
1 to 100
58
XQuery and JSONiq: Basic Operations
59
Basic Operations: Arithmetics
addition 1 + 1
substraction 42 - 10
multiplication 6 * 7
division 42.3 div 7.2
integer division 42 idiv 9
module 42 mod 9
60
Basic Operations: Atomization (XML only)
<a>42</a> + 1
61
Basic Operations: Atomization (XML only)
<a>42</a> + 1
42 + 1
62
Basic Operations: Atomization (XML only)
<a>42</a> + 1
42 + 1
43
63
Basic Operations: Empty sequence
() + 1
64
Basic Operations: Empty sequence
()
() + 1
65
Basic Operations: types
"foo" + 2
66
Basic Operations: types
"foo" + 2ERROR
67
Basic Operations: cardinality
(3, 4) + 2
68
Basic Operations: cardinality
(3, 4) + 2ERROR
69
Basic Operations: Strings
concatenation "foo" || "bar"concat("foo", "bar")string-join(
("foo", "bar", "foobar"),"-"
)
sub-string substr("foobar", 4, 3)
length string-length("foobar")
70
Basic Operations: Value Comparison
equality 1 + 1 eq 2
inequality 6 * 7 ne 21 * 2
greater than 234 gt 123234 ge 123
less than 42.3 lt 7.242.3 le 7.2
71
Basic Operations:Value Comparison
1 eq 2"foo" ne "bar1 lt 22 gt 32 le 24 ge 3
72
Basic Operations:Value Comparison
1 eq 2"foo" ne "bar1 lt 22 gt 32 le 24 ge 3
73
Basic Operations:Value Comparison
() le 2
74
Basic Operations:Value Comparison
() le 2
()
75
Basic Operations:Value Comparison
"foo" le 2
76
Basic Operations:Value Comparison
"foo" le 2ERROR
77
Basic Operations:General Comparison
(1, 2, 3, 4, 5) = 1
78
Basic Operations:General Comparison
(1, 2, 3, 4, 5) = 1
=
79
Basic Operations:General Comparison
(1, 2, 3, 4, 5) < (2, 3, 4, 5, 6)
80
Basic Operations:General Comparison
(1, 2, 3, 4, 5) < (2, 3, 4, 5, 6)
<
81
Basic Operations:General Comparison
(1, 2, 3, 4, 5) >= (6, 7, 8, 9, 10)
82
Basic Operations:General Comparison
(1, 2, 3, 4, 5) >= (6, 7, 8, 9, 10)
83
Basic Operations:General Comparison
(1, 2, 3, 4, 5) >= ("foo", 7, 8, 9, 10)
84
Basic Operations:General Comparison
(1, 2, 3, 4, 5) >= ("foo", 7, 8, 9, 10)ERROR
85
Basic Operations:General Comparison
<?xml version="1.0" encoding="UTF-8"?> <countries> <country code="CH"> <name>Switzerland</name> </country> <country code="F"> <name>France</name> </country> <country code="D"> <name>Germany</name> </country> <country code="I"> <name>Italy</name> </country> <country code="A"> <name>Austria</name> </country> </countries>
doc("myfile.xml")//name = "Switzerland"
86
Basic Operations:General Comparison
<?xml version="1.0" encoding="UTF-8"?> <countries> <country code="CH"> <name>Switzerland</name> </country> <country code="F"> <name>France</name> </country> <country code="D"> <name>Germany</name> </country> <country code="I"> <name>Italy</name> </country> <country code="A"> <name>Austria</name> </country> </countries>
doc("myfile.xml")//name = "Switzerland"
87
Basic Operations:General Comparison
<?xml version="1.0" encoding="UTF-8"?> <countries> <country code="CH"> <name>Switzerland</name> </country> <country code="F"> <name>France</name> </country> <country code="D"> <name>Germany</name> </country> <country code="I"> <name>Italy</name> </country> <country code="A"> <name>Austria</name> </country> </countries>
doc("myfile.xml")//name = "Switzerland"
88
Basic Operations:General Comparison
collection("countries").name = "Switzerland"
{"name" : "Switzerland","code" : "CH"
}{
"name" : "France","code" : "F"
}{
"name" : "Germany","code" : "D"
}{
"name" : "Italy","code" : "I"
}{
"name" : "Austria","code" : "A"
}
89
Basic Operations: Logics
conjunction 1+1 eq 2 and 2+2 eq 4
disjunction 1+1 eq 2 or 2+2 eq 4
not not(100 mod 5 eq 0)
90
Two-valued logics (JSONiq)
true and false
true or true
not false
not(true or not(false and true))
91
Two-valued logics (XQuery)
true() and false()
true() or true()
not(false())
not(true() or not(false() and true()))
92
Non-booleans
if("")...
93
Non-booleans
if("foo")...
94
Non-booleans
if(0)...
95
Non-booleans
if(42)...
96
Non-booleans
if(())...
97
Non-booleans
if((<foo/>, <bar/>, 1, "foo"))...
98
Non-booleans
if(({ "foo" : "bar" }, 1, "foo"))...
99
Composability
100
Rule of Thumb
Any Expressioncan be the operand of any other expression.
101
Composability
(1 + ((<b><a>{2+2}</a></b>)/a)) to 50
102
Composability
(1 + ((<b><a>{4}</a></b>)/a)) to 50
103
Composability
(1 + ((<b><a>4</a></b>)/a)) to 50
104
Composability
(1 + (<a>4</a>)) to 50
105
Composability
(1 + 4) to 50
106
Composability
5 to 50
107
Composability
5, 6, 7, ..., 49, 50
108
Composability
(1 + (({"b":[{"a": 2+2}]}).b[].a)) to 50
109
PrecedencePrecedence (low first)CommaData Flow (FLWOR, if-then-else, switch...)LogicComparisonString concatenationRangeArithmeticPath expressionsFilter predicates, dynamic function callsLiterals, constructors and variablesFunction calls, named function references, inline functions
110
PrecedencePrecedence (low first)CommaData Flow (FLWOR, if-then-else, switch...)LogicComparisonString concatenationRangeArithmeticPath expressionsFilter predicates, dynamic function callsLiterals, constructors and variablesFunction calls, named function references, inline functions
Use parentheses to override or when in doubt!
111
XML Construction reloaded<foo attr="{string-length("foobar")}">{string-join(for $i in 1 to 10 return string($i),"-"
)}</foo>
112
XML Construction reloaded<foo attr="{string-length("foobar")}">{string-join(for $i in 1 to 10 return string($i),"-"
)}</foo>
<foo attr="6">1-2-3-4-5-6-7-8-9-10</foo>
113
JSON Construction reloaded{"attr" : string-length("foobar")"values" : [for $i in 1 to 10return string($i)
]}
114
JSON Construction reloaded{"attr" : string-length("foobar")"values" : [for $i in 1 to 10return string($i)
]}
{"attr" : 6,"values" : [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
}
115
Data Flow(XQuery, JSONiq)
116
Conditional Expressions
if(count(doc("file.xml")//country) gt 1000)then "Large file!"else "Small file."
117
Switch Expressions
switch($country/code)case "CH" return("gsw", "de", "fr", "it", "rm")
case "F" return "fr"case "D" return "de"case "I" return "it"default return "en"
118
Try Catch Expressions
try {xs:integer($country/code)
} catch * {"A country code is not an integer!"
}
119
FLWOR Expressions(XQuery, JSONiq)
Alexander Raths / 123RF Stock Photo
120
Let clauses
let $x := 2return $x * $x
121
For clauses
for $x in (1, 2, 4)return $x * $x
122
Where clauses
for $x in 1 to 10where $x - 2 gt 5return $x * $x
123
For clauses
for $x in 1 to 10return{"number": $x,"square": $x * $x
}
124
Order by clauses
for $x in collection("countries")order by $x.populationreturn $x.name
125
Group by clauses{
"continents" : [for $x in collection("countries")group by $continent := $x.continentreturn{
"code": $continent,"countries" : [
for $country in $xreturn $country.name
]}
]}
126
Tuple stream semantics
for $x in 1 to 10
where $x - 2 gt 5
return $x * $x
127
Tuple stream semantics
for $x in 1 to 10
where $x - 2 gt 5
return $x * $x
x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9 x 10
128
Tuple stream semantics
for $x in 1 to 10
where $x - 2 gt 5
return $x * $x
x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9 x 10
x 8 x 9 x 10
129
Tuple stream semantics
for $x in 1 to 10
where $x - 2 gt 5
return $x * $x
x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9 x 10
x 8 x 9 x 10
(64, 81, 100)
130
Tuple stream semantics
for $x in 1 to 5
let $y := $x - 2
return $x * $y
131
Tuple stream semantics
for $x in 1 to 5
let $y := $x - 2
return $x * $y
x 1 x 2 x 3 x 4 x 5
132
Tuple stream semantics
for $x in 1 to 5
let $y := $x - 2
return $x * $y
x 1 x 2 x 3 x 4 x 5
x 1 x 2 x 3 x 4 x 5y -1 y 0 y 1 y 2 y 3
133
Tuple stream semantics
for $x in 1 to 5
let $y := $x - 2
return $x * $y
x 1 x 2 x 3 x 4 x 5
x 1 x 2 x 3 x 4 x 5y -1 y 0 y 1 y 2 y 3
(-1, 0, 3, 8, 15)
134
Tuple stream semanticsfor $x in 1 to 5
group by $y := $x mod 2
return {"foo" : $y,"bar" : count($x)
}
135
Tuple stream semanticsfor $x in 1 to 5
group by $y := $x mod 2
return {"foo" : $y,"bar" : count($x)
}
x 1 x 2 x 3 x 4 x 5
136
Tuple stream semanticsfor $x in 1 to 5
group by $y := $x mod 2
return {"foo" : $y,"bar" : count($x)
}
x 1 x 2 x 3 x 4 x 5
y 0 y 1
137
Tuple stream semanticsfor $x in 1 to 5
group by $y := $x mod 2
return {"foo" : $y,"bar" : count($x)
}
x 1 x 2 x 3 x 4 x 5
y 0 x 2 4 y 1 x 1 3 5
138
Tuple stream semanticsfor $x in 1 to 5
group by $y := $x mod 2
return {"foo" : $y,"bar" : count($x)
}
x 1 x 2 x 3 x 4 x 5
y 0 x 2 4 y 1 x 1 3 5
{ "foo" : 0, "bar" : 2 }, { "foo" : 1, "bar" : 3 }
139
Types
140
Types
§ Simple types: by name
integer
141
Types
§ Simple types: by name
integer§ Complex types: by kind
object
142
Cardinality (JSON/JSONiq)
integerboolean?array+object*
143
Cardinality (XML/XQuery)
xs:integerxs:boolean?element(foo)+
document-node()*
144
Type usage
let $x as integer := 2return $x + $x
145
Type usage
let $x as object :={ "foo" : 1, "bar" : 2 }
return $x.foo + $x.bar
146
Type usage
for $x as array in ([ 1 ],[ 2 ]
)return $x[] + $x[]
147
Type usage (XQuery)
let $x as node() :=<foo>2</foo>
return $x + $x
148
Type usage
for $x as array* in ([ 1 ],[ 2 ]
)return $x + $x treat as integer
149
Type usage
for $x as array* in ([ 1 ],[ 2 ]
)return $x + $x cast as double
150
Type usage
for $x as array* in ([ 1 ],[ 2 ]
)return double($x + $x)
151
Type usage
3.14 instance of integer
152
Type usage
3.14 castable as double
153
Type usage
declare function local:is-big-data($threshold as integer,$objects as object*
) as boolean{count($objects) gt $threshold
};
154
Type usage
declare function local:is-big-data($threshold as integer,$objects as object*
) as boolean{count($objects) gt $threshold
};
local:is-big-data(1000, collection("countries"))
155
Validationimport'schema'namespace'm!=!http://www.w3.org/1998/Math/MathML!!!at!"http://www.w3.org/Math/XMLSchema/mathml3/mathml3.xsd";!!validate!{!<m:math!xmlns:m="http://www.w3.org/1998/Math/MathML">!!!<m:apply>!!!!!<m:eq/>!!!!!<m:ci>!!!!!!!x!!!!!</m:ci>!!!!!<m:apply>!!!!!!!<m:root/>!!!!!!!<m:cn>!!!!!!!!!2!!!!!!!</m:cn>!!!!!</m:apply>!!!</m:apply>!!</m:math>!}!!
156
Not covered
§ Advanced XPath (kind tests, axes, ...)§ Higher-order functions§ Count clause§ Window clause§ Node comparison§ Errors§ Typeswitch
157
There’s more
Updates
XQuery Full-Text
Scripting
insert node <foo/> into $doc/bar
/books/book[@number="1"]/titlecontains text "improve" using stemming
while ($x gt 0) { $x := $x – 1; }
158
Architecture of a query processing engine
159
Parsing
QueryAbstract Syntax Tree
count(for $i in json-lines("file.json")return $i.field
)
"file.json"
json-lines() .field
$i
returnfor
FLWOR
count()
$i
160
Translating
Abstract Syntax Tree
"file.json"
json-lines() .field
$i
returnfor
FLWOR
count()
$i
Expression Tree
StringLiteralExpression
JSONLinesFunctionCall ObjectLookupExpression
VariableExpression
ReturnClause
ForClause
FLWORExpression
CountFunctionCall
VariableExpression
StringLiteralExpression
161
Optimization
Expression Tree
StringLiteralExpression
JSONLinesFunctionCall ObjectLookupExpression
VariableExpression
ReturnClause
ForClause
FLWORExpression
CountFunctionCall
VariableExpression
StringLiteralExpression
162
Code Generation
Expression Tree
StringLiteralExpression
JSONFileFunctionCall ObjectLookupExpression
VariableExpression
ReturnClause
ForClause
FLWORExpression
CountFunctionCall
VariableExpression
StringLiteralExpressionStringIterator
JSONFileIterator
ObjectLookupExpression
VariableExpression
ReturnIterator
ForIterator
CountIterator
VariableName
StringLiteralExpression
Iterator Tree
163
Execution
StringIterator
JSONFileIterator
ObjectLookupExpression
VariableExpression
ReturnIterator
ForIterator
CountIterator
VariableName
StringLiteralExpression
Sequence of items
Stream of tuples
164
Materialized execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples
165
Materialized execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples
166
Materialized execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
$i: { "field" : 1 }ReturnIterator
ForIterator
Stream of tuples
167
Materialized execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
$i: { "field" : 1 }$i: { "field" : 6 }
ReturnIterator
ForIterator
Stream of tuples
168
Materialized execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }
ReturnIterator
ForIterator
Stream of tuples
169
Materialized execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }
ReturnIterator
ForIterator
Stream of tuples
170
Materialized execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples
171
Materialized execution
ReturnIterator
ForIterator
Stream of tuples
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
16432
172
Materialized execution
ReturnIterator
ForIterator
Stream of tuples
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
16432
Can take lots of memory!
173
Streamed execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples
174
Streamed execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples $i: { "field" : 1 }
175
Streamed execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples
1
176
Streamed execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples $i: { "field" : 6 }
177
Streamed execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples
6
178
Streamed execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples $i: { "field" : 4 }
179
Streamed execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples
4
180
Streamed execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples $i: { "field" : 3 }
181
Streamed execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples
3
182
Streamed execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples $i: { "field" : 2 }
183
Streamed execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples
2
184
Streamed execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples
2
Can take lots of time!
185
Parallel execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples
186
Parallel execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples
$i: { "field" : 1 }$i: { "field" : 6 }
$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
187
Parallel execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples
$i: { "field" : 1 }$i: { "field" : 6 }
$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
16
432
188
Parallel execution
$i: { "field" : 1 }$i: { "field" : 6 }$i: { "field" : 4 }$i: { "field" : 3 }$i: { "field" : 2 }
ReturnIterator
ForIterator
Stream of tuples
Can take lots of machines!
