Post on 24-Jul-2020
transcript
Neues inOpen-Source-SQL-Datenbanken
@MarkusWinand • @ModernSQL
http://www.almaden.ibm.com/cs/people/chamberlin/sequel-1974.pdf
Neues inOpen-Source-SQL-Datenbanken
@MarkusWinand • @ModernSQL
http://www.almaden.ibm.com/cs/people/chamberlin/sequel-1974.pdf
SQL-92
CHECKConstraints
CHECK Constraints Since SQL-92CREATETABLEorder_lines(…qtyINTEGERNOTNULLCHECK(qty>0),…)
CHECK Constraints Since SQL-92CREATETABLEorder_lines(…qtyINTEGERNOTNULLCHECK(qty>0),…)
INSERT…(…,qty,…)VALUES(…,1,…)
CHECK Constraints Since SQL-92CREATETABLEorder_lines(…qtyINTEGERNOTNULLCHECK(qty>0),…)
INSERT…(…,qty,…)VALUES(…,1,…)INSERT…(…,qty,…)VALUES(…,3,…)
CHECK Constraints Since SQL-92CREATETABLEorder_lines(…qtyINTEGERNOTNULLCHECK(qty>0),…)
INSERT…(…,qty,…)VALUES(…,1,…)INSERT…(…,qty,…)VALUES(…,3,…)INSERT…(…,qty,…)VALUES(…,0,…)
CHECK Constraints Since SQL-92CREATETABLEorder_lines(…qtyINTEGERNOTNULLCHECK(qty>0),…)
INSERT…(…,qty,…)VALUES(…,1,…)INSERT…(…,qty,…)VALUES(…,3,…)INSERT…(…,qty,…)VALUES(…,0,…)
Before MySQL 8.0.16and MariaDB 10.2:
Syntax accepted,Constraint ignored
CHECK Constraints Since SQL-921999
2001
2003
2005
2007
2009
2011
2013
2015
2017
10.2 MariaDB8.0.16 MySQL
8.3 PostgreSQL3.5.7 SQLite
9.7 DB2 LUW11gR1 Oracle
2008R2 SQL Server
INTERSECT and EXCEPT
Since SQL-92INTERSECT & EXCEPT
UNION[ALL]
Concatenatestwo results
Since SQL-92INTERSECT & EXCEPT
UNION[ALL]
Concatenatestwo results
INTERSECT[ALL]
Common rows from two results
Since SQL-92INTERSECT & EXCEPT
UNION[ALL]
Concatenatestwo results
INTERSECT[ALL]
Common rows from two results
EXCEPT[ALL]
Remove rowsfrom first result
INTERSECT & EXCEPT Since SQL-921999
2001
2003
2005
2007
2009
2011
2013
2015
2017
10.3[0] MariaDBMySQL
8.3 PostgreSQL3.5.7[0] SQLite
9.7 DB2 LUW11gR1[0] Oracle
2008R2[0] SQL Server[0]Not [all]
SQL:1999
LATERAL
Select-list sub-queries must be scalar[0]:
LATERAL Before SQL:1999
SELECT…,(SELECTcolumn_1FROMt1WHEREt1.x=t2.y)AScFROMt2…
(an atomic quantity that can hold only one value at a time[1])
[0] Neglecting row values and other workarounds here; [1] https://en.wikipedia.org/wiki/Scalar
Select-list sub-queries must be scalar[0]:
LATERAL Before SQL:1999
SELECT…,(SELECTcolumn_1FROMt1WHEREt1.x=t2.y)AScFROMt2…
(an atomic quantity that can hold only one value at a time[1])
[0] Neglecting row values and other workarounds here; [1] https://en.wikipedia.org/wiki/Scalar
✗,column_2
More thanone column? ⇒Syntax error
Select-list sub-queries must be scalar[0]:
LATERAL Before SQL:1999
SELECT…,(SELECTcolumn_1FROMt1WHEREt1.x=t2.y)AScFROMt2…
(an atomic quantity that can hold only one value at a time[1])
[0] Neglecting row values and other workarounds here; [1] https://en.wikipedia.org/wiki/Scalar
✗,column_2
More thanone column? ⇒Syntax error
}More thanone row?
⇒Runtime error!
SELECT*FROMt1JOIN(SELECT*FROMt2WHEREt2.x=t1.x)derived_tableON(derived_table.x=t1.x)
Derived tables (from clause subqueries) cannot look outside:
LATERAL Before SQL:1999
SELECT*FROMt1JOIN(SELECT*FROMt2WHEREt2.x=t1.x)derived_tableON(derived_table.x=t1.x)
Derived tables (from clause subqueries) cannot look outside:
LATERAL Before SQL:1999
Invalid
SELECT*FROMt1JOIN(SELECT*FROMt2WHEREt2.x=t1.x)derived_tableON(derived_table.x=t1.x)
Derived tables (from clause subqueries) cannot look outside:
LATERAL Before SQL:1999
Belongs
there
SELECT*FROMt1CROSSJOINLATERAL(SELECT*FROMt2WHEREt2.x=t1.x)derived_tableON(true)
LATERAL Since SQL:1999Lateral derived queries can see table names defined before:
SELECT*FROMt1CROSSJOINLATERAL(SELECT*FROMt2WHEREt2.x=t1.x)derived_tableON(true)
LATERAL Since SQL:1999
Valid due to
LATERAL
keyword
Lateral derived queries can see table names defined before:
SELECT*FROMt1CROSSJOINLATERAL(SELECT*FROMt2WHEREt2.x=t1.x)derived_tableON(true)
LATERAL Since SQL:1999
Valid due to
LATERAL
keyword
Useless, but still required
Lateral derived queries can see table names defined before:
SELECT*FROMt1CROSSJOINLATERAL(SELECT*FROMt2WHEREt2.x=t1.x)derived_tableON(true)
LATERAL Since SQL:1999
Valid due to
LATERAL
keyword
Lateral derived queries can see table names defined before:
Use CROSS JOIN to omit the ON clause
But WHY?
Use-CasesLATERAL
‣ Top-N per group
inside a lateral derived tableFETCHFIRST (or LIMIT, TOP)applies per row from left tables.
Use-CasesLATERAL
FROMtCROSSJOINLATERAL(SELECT…FROM…WHEREt.c=…ORDERBY…LIMIT10)derived_table
‣ Top-N per group
inside a lateral derived tableFETCHFIRST (or LIMIT, TOP)applies per row from left tables.
Use-CasesLATERAL
FROMtCROSSJOINLATERAL(SELECT…FROM…WHEREt.c=…ORDERBY…LIMIT10)derived_table
‣ Top-N per group
inside a lateral derived tableFETCHFIRST (or LIMIT, TOP)applies per row from left tables.
Use-CasesLATERAL
Add proper indexfor Top-N query
https://use-the-index-luke.com/sql/partial-results/top-n-queries
FROMtCROSSJOINLATERAL(SELECT…FROM…WHEREt.c=…ORDERBY…LIMIT10)derived_table
‣ Top-N per group
inside a lateral derived tableFETCHFIRST (or LIMIT, TOP)applies per row from left tables.
‣ Also useful to find most recent news from several subscribed topics (“multi-source top-N”).
Use-CasesLATERAL
FROMtCROSSJOINLATERAL(SELECT…FROM…WHEREt.c=…ORDERBY…LIMIT10)derived_table
‣ Top-N per group
inside a lateral derived tableFETCHFIRST (or LIMIT, TOP)applies per row from left tables.
‣ Also useful to find most recent news from several subscribed topics (“multi-source top-N”).
‣ Table function arguments
(TABLE often implies LATERAL)
Use-CasesLATERAL
FROMtJOINTABLE(your_func(t.c))
LATERAL Availability1999
2001
2003
2005
2007
2009
2011
2013
2015
2017
5.1 MariaDB8.0.14 MySQL
9.3 PostgreSQLSQLite
9.1 DB2 LUW11gR1[0] 12cR1 Oracle
2005[1] SQL Server[0]Undocumented. Requires setting trace event 22829.[1]LATERAL is not supported as of SQL Server 2016 but [CROSS|OUTER]APPLY can be used for the same effect.
WITH(Common Table Expressions)
WITH (non-recursive) The ProblemNested queries are hard to read:
SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)
Understand
this first
WITH (non-recursive) The ProblemNested queries are hard to read:
SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)
Then this...
WITH (non-recursive) The ProblemNested queries are hard to read:
SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)
Then this...
WITH (non-recursive) The ProblemNested queries are hard to read:
SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)
Finally the first line makes sense
WITH (non-recursive) The ProblemNested queries are hard to read:
SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)
WITH (non-recursive) Since SQL:1999
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)
Keyword
WITH (non-recursive) Since SQL:1999
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)
Name of CTE and (here optional) column names
WITH (non-recursive) Since SQL:1999
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)
Definition
WITH (non-recursive) Since SQL:1999
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)
Introduces another CTE
Don't repeat WITH
WITH (non-recursive) Since SQL:1999
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)
WITH (non-recursive) Since SQL:1999
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)
May refer toprevious CTEs
WITH (non-recursive) Since SQL:1999
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)AS(SELECT…FROM…)
SELECT…FROMbJOINcON(…)
Third CTE
WITH (non-recursive) Since SQL:1999
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)AS(SELECT…FROM…)
SELECT…FROMbJOINcON(…)
No comma!
WITH (non-recursive) Since SQL:1999
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)AS(SELECT…FROM…)
SELECT…FROMbJOINcON(…)
Main query
WITH (non-recursive) Since SQL:1999
CTEs are statement-scoped views:
WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),
b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),
c(…)AS(SELECT…FROM…)
SELECT…FROMbJOINcON(…)
Read top down
WITH (non-recursive) Since SQL:1999
PostgreSQL “issues”WITH (non-recursive)
In pre 12 PostgreSQL WITH queries are “optimizer fences”:
WITHcteAS(SELECT*FROMnews)SELECT*FROMcteWHEREtopic=1
CTEScanoncte(rows=6370)Filter:topic=1CTEcte->SeqScanonnews(rows=10000001)
PostgreSQL “issues”WITH (non-recursive)
In pre 12 PostgreSQL WITH queries are “optimizer fences”:
WITHcteAS(SELECT*FROMnews)SELECT*FROMcteWHEREtopic=1
CTEScanoncte(rows=6370)Filter:topic=1CTEcte->SeqScanonnews(rows=10000001)
PostgreSQL “issues”WITH (non-recursive)
In pre 12 PostgreSQL WITH queries are “optimizer fences”:
WITHcteAS(SELECT*FROMnews)SELECT*FROMcteWHEREtopic=1
CTEScanoncte(rows=6370)Filter:topic=1CTEcte->SeqScanonnews(rows=10000001)
PostgreSQL “issues”WITH (non-recursive)
CTE doesn't
know about the outer
filter
In pre 12 PostgreSQL WITH queries are “optimizer fences”:
WITHcteAS(SELECT*FROMnews)SELECT*FROMcteWHEREtopic=1
Views and derived tables support "predicate pushdown":SELECT*FROM(SELECT*FROMnews)nWHEREtopic=1;
PostgreSQL “issues”WITH (non-recursive)
Views and derived tables support "predicate pushdown":SELECT*FROM(SELECT*FROMnews)nWHEREtopic=1;
BitmapHeapScanonnews(rows=6370)->BitmapIndexScanonidx(rows=6370)Cond:topic=1
PostgreSQL “issues”WITH (non-recursive)
Views and derived tables support "predicate pushdown":SELECT*FROM(SELECT*FROMnews)nWHEREtopic=1;
BitmapHeapScanonnews(rows=6370)->BitmapIndexScanonidx(rows=6370)Cond:topic=1
PostgreSQL “issues”WITH (non-recursive)
PostgreSQL 12 inlines non-recursive CTEs if it isreferred to only once.
Use [NOT]MATERIALIZED to force it (not SQL standard).
WITHcteAS[NOT]MATERIALIZED(SELECT*FROMnews)SELECT*FROMcteWHEREtopic=1
PostgreSQL “issues”WITH (non-recursive)
AvailabilityWITH (non-recursive)1999
2001
2003
2005
2007
2009
2011
2013
2015
2017
5.1 10.2 MariaDB8.0 MySQL
8.4 PostgreSQL3.8.3[0] SQLite
7.0 DB2 LUW9iR2 Oracle
2005[1] SQL Server[0]Only for top-level SELECT statements[1]Only allowed at the very begin of a statement. E.g. WITH...INSERT...SELECT.
SQL:2003
BOOLEANTests
Before we start
SQL uses a three-valued logic. Boolean values are either
true, false or unknown(=null).
See: http://modern-sql.com/concept/three-valued-logic
BOOLEANAggregates
BOOLEANTests
Similar to isnull, there are tests for each Boolean value(of which there are three: true, false, unknown/null)
IS[NOT][TRUE|FALSE|UNKNOWN]
Since SQL:2003
CREATETABLEprices(…valid_fromDATENOTNULL,valid_toDATE,--null:openend…CHECK(valid_from<valid_to),);
WHEREvalid_from<CURRENT_DATEAND(valid_to<=CURRENT_DATE)ISNOTFALSE
BOOLEANTests Since SQL:2003
CREATETABLEprices(…valid_fromDATENOTNULL,valid_toDATE,--null:openend…CHECK(valid_from<valid_to),);
WHEREvalid_from<CURRENT_DATEAND(valid_to<=CURRENT_DATE)ISNOTFALSE
BOOLEANTests Since SQL:2003
CREATETABLEprices(…valid_fromDATENOTNULL,valid_toDATE,--null:openend…CHECK(valid_from<valid_to),);
WHEREvalid_from<CURRENT_DATEAND(valid_to<=CURRENT_DATE)ISNOTFALSE
BOOLEANTests Since SQL:2003
UNKNOWN ifVALID_TO is NULL
CREATETABLEprices(…valid_fromDATENOTNULL,valid_toDATE,--null:openend…CHECK(valid_from<valid_to),);
WHEREvalid_from<CURRENT_DATEAND(valid_to<=CURRENT_DATE)ISNOTFALSE
BOOLEANTests Since SQL:2003
UNKNOWN ifVALID_TO is NULL Takes TRUE and
UNKNOWN
BOOLEANTests Since SQL:20031999
2001
2003
2005
2007
2009
2011
2013
2015
2017
5.1 MariaDB5.0.51a MySQL8.3 PostgreSQL
3.23.0[0] SQLite
DB2 LUWOracleSQL Server
[0]No IS [NOT] UNKNOWN. Use IS [NOT] NULL instead.
OVERand
PARTITIONBY
OVER (PARTITION BY) The ProblemTwo distinct concepts could not be used independently:
OVER (PARTITION BY) The ProblemTwo distinct concepts could not be used independently:
‣Merge rows with the same key properties
‣ GROUPBY to specify key properties
‣ DISTINCT to use full row as key
OVER (PARTITION BY) The ProblemTwo distinct concepts could not be used independently:
‣Merge rows with the same key properties
‣ GROUPBY to specify key properties
‣ DISTINCT to use full row as key
‣ Aggregate data from related rows ‣ Requires GROUPBY to segregate the rows
‣ COUNT, SUM, AVG, MIN, MAX to aggregate grouped rows
OVER (PARTITION BY) The Problem
OVER (PARTITION BY) The Problem
SELECTc1,c2FROMt
SELECTc1,c2FROMt
OVER (PARTITION BY) The Problem
Yes ⇠
Mer
ge ro
ws ⇢
No SELECTc1
,c2FROMt
SELECTc1,c2FROMt
OVER (PARTITION BY) The Problem
Yes ⇠
Mer
ge ro
ws ⇢
No SELECTc1
,c2FROMt
SELECTDISTINCTc1,c2FROMt
SELECTc1,c2FROMt
OVER (PARTITION BY) The Problem
Yes ⇠
Mer
ge ro
ws ⇢
No
No ⇠ Aggregate ⇢ Yes
SELECTc1,c2FROMt
SELECTDISTINCTc1,c2FROMt
SELECTc1,c2FROMt
SELECTc1,SUM(c2)totFROMtGROUPBYc1
OVER (PARTITION BY) The Problem
Yes ⇠
Mer
ge ro
ws ⇢
No
No ⇠ Aggregate ⇢ Yes
SELECTc1,c2FROMt
SELECTDISTINCTc1,c2FROMt
SELECTc1,c2FROMt
SELECTc1,SUM(c2)totFROMtGROUPBYc1
SELECTc1,SUM(c2)totFROMtGROUPBYc1
OVER (PARTITION BY) The Problem
Yes ⇠
Mer
ge ro
ws ⇢
No
No ⇠ Aggregate ⇢ Yes
SELECTc1,c2FROMt
SELECTDISTINCTc1,c2FROMt
SELECTc1,c2FROMt
SELECTc1,SUM(c2)totFROMtGROUPBYc1
SELECTc1,SUM(c2)totFROMtGROUPBYc1
OVER (PARTITION BY) The Problem
Yes ⇠
Mer
ge ro
ws ⇢
No
No ⇠ Aggregate ⇢ Yes
SELECTc1,c2FROMt
SELECTDISTINCTc1,c2FROMt
SELECTc1,c2FROMtJOIN()taON(t.c1=ta.c1)
SELECTc1,SUM(c2)totFROMtGROUPBYc1
SELECTc1,SUM(c2)totFROMtGROUPBYc1
OVER (PARTITION BY) The Problem
Yes ⇠
Mer
ge ro
ws ⇢
No
No ⇠ Aggregate ⇢ Yes
SELECTc1,c2FROMt
SELECTDISTINCTc1,c2FROMt
SELECTc1,c2FROMtJOIN()taON(t.c1=ta.c1)
SELECTc1,SUM(c2)totFROMtGROUPBYc1
SELECTc1,SUM(c2)totFROMtGROUPBYc1
OVER (PARTITION BY) The Problem
Yes ⇠
Mer
ge ro
ws ⇢
No
No ⇠ Aggregate ⇢ Yes
SELECTc1,c2FROMt
SELECTDISTINCTc1,c2FROMt
SELECTc1,c2FROMtJOIN()taON(t.c1=ta.c1)
SELECTc1,SUM(c2)totFROMtGROUPBYc1
,tot
SELECTc1,SUM(c2)totFROMtGROUPBYc1
OVER (PARTITION BY) The Problem
Yes ⇠
Mer
ge ro
ws ⇢
No
No ⇠ Aggregate ⇢ Yes
SELECTc1,c2FROMt
SELECTDISTINCTc1,c2FROMt
SELECTc1,c2FROMtJOIN()taON(t.c1=ta.c1)
SELECTc1,SUM(c2)totFROMtGROUPBYc1
,tot
SELECTc1,SUM(c2)totFROMtGROUPBYc1
OVER (PARTITION BY) Since SQL:2003
Yes ⇠
Mer
ge ro
ws ⇢
No
No ⇠ Aggregate ⇢ Yes
SELECTc1,c2FROMt
SELECTDISTINCTc1,c2FROMt
SELECTc1,c2FROMtFROMt
SELECTc1,SUM(c2)totFROMtGROUPBYc1
OVER (PARTITION BY) Since SQL:2003
Yes ⇠
Mer
ge ro
ws ⇢
No
No ⇠ Aggregate ⇢ Yes
SELECTc1,c2FROMt
SELECTDISTINCTc1,c2FROMt
SELECTc1,c2FROMt
FROMt
,SUM(c2)OVER(PARTITIONBYc1)
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000
OVER (PARTITION BY) How it works
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000
OVER (PARTITION BY) How it works
Look here
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000
OVER (PARTITION BY) How it works
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000
OVER (PARTITION BY) How it works
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000
OVER (PARTITION BY) How it works
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000
OVER (PARTITION BY) How it works
dep salary1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000
SELECTdep,salary,SUM(salary)OVER()FROMemp
OVER (PARTITION BY) How it works
)
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary1 1000 100022 1000 200022 1000 2000333 1000 3000333 1000 3000333 1000 3000
OVER (PARTITION BY) How it works
)
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary1 1000 100022 1000 200022 1000 2000333 1000 3000333 1000 3000333 1000 3000
OVER (PARTITION BY) How it works
)PARTITIONBYdep
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary1 1000 100022 1000 200022 1000 2000333 1000 3000333 1000 3000333 1000 3000
OVER (PARTITION BY) How it works
)PARTITIONBYdep
SELECTdep,salary,SUM(salary)OVER()FROMemp
dep salary1 1000 100022 1000 200022 1000 2000333 1000 3000333 1000 3000333 1000 3000
OVER (PARTITION BY) How it works
)PARTITIONBYdep
OVERand
ORDERBY(Framing & Ranking)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
OVER (ORDER BY) The Problem
SELECTid,value,FROMtransactionst
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
OVER (ORDER BY) The Problem
SELECTid,value,FROMtransactionst
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
OVER (ORDER BY) The Problem
SELECTid,value,FROMtransactionst
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
OVER (ORDER BY) The Problem
SELECTid,value,
(SELECTSUM(value)FROMtransactionst2WHEREt2.id<=t.id)
FROMtransactionst
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
OVER (ORDER BY) The Problem
SELECTid,value,
(SELECTSUM(value)FROMtransactionst2WHEREt2.id<=t.id)
FROMtransactionst
Range segregation (<=)not possible with
GROUP BY orPARTITION BY
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
(SELECTSUM(value)FROMtransactionst2WHEREt2.id<=t.id)
FROMtransactionst
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYid
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDING
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +30
22 3 -10 +20
333 4 +50 +70
333 5 -30 +40
333 6 -20 +20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10
22 2 +20
22 3 -10
333 4 +50
333 5 -30
333 6 -20
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
OVER (ORDER BY) Since SQL:2003
SELECTid,value,
FROMtransactionst
SUM(value)OVER(
)
acnt id value balance
1 1 +10 +10
22 2 +20 +20
22 3 -10 +10
333 4 +50 +50
333 5 -30 +20
333 6 -20 .0
ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW
PARTITIONBYacnt
OVER (ORDER BY) Since SQL:2003With OVER(ORDERBYn) a new type of functions make sense:
n ROW_NUMBER RANK DENSE_RANK PERCENT_RANK CUME_DISTA 1 1 1 0 0.25B 2 2 2 0.33… 0.75B 3 2 2 0.33… 0.75X 4 4 3 1 1
‣ Aggregates without GROUPBY
Use CasesOVER (SQL:2003)
‣ Aggregates without GROUPBY
‣ Running totals, moving averages
Use Cases
AVG(…)OVER(ORDERBY…ROWSBETWEEN3PRECEDINGAND3FOLLOWING)moving_avg
OVER (SQL:2003)
‣ Aggregates without GROUPBY
‣ Running totals, moving averages
‣ Ranking
Use Cases
AVG(…)OVER(ORDERBY…ROWSBETWEEN3PRECEDINGAND3FOLLOWING)moving_avg
OVER (SQL:2003)
‣ Aggregates without GROUPBY
‣ Running totals, moving averages
‣ Ranking‣ Top-N per Group
Use Cases
SELECT*FROM(SELECTROW_NUMBER()OVER(PARTITIONBY…ORDERBY…)rn,t.*FROMt)numbered_tWHERErn<=3
AVG(…)OVER(ORDERBY…ROWSBETWEEN3PRECEDINGAND3FOLLOWING)moving_avg
OVER (SQL:2003)
‣ Aggregates without GROUPBY
‣ Running totals, moving averages
‣ Ranking‣ Top-N per Group
‣ Avoiding self-joins
Use Cases
SELECT*FROM(SELECTROW_NUMBER()OVER(PARTITIONBY…ORDERBY…)rn,t.*FROMt)numbered_tWHERErn<=3
AVG(…)OVER(ORDERBY…ROWSBETWEEN3PRECEDINGAND3FOLLOWING)moving_avg
OVER (SQL:2003)
‣ Aggregates without GROUPBY
‣ Running totals, moving averages
‣ Ranking‣ Top-N per Group
‣ Avoiding self-joins
[… many more …]
Use Cases
SELECT*FROM(SELECTROW_NUMBER()OVER(PARTITIONBY…ORDERBY…)rn,t.*FROMt)numbered_tWHERErn<=3
AVG(…)OVER(ORDERBY…ROWSBETWEEN3PRECEDINGAND3FOLLOWING)moving_avg
OVER (SQL:2003)
1999
2001
2003
2005
2007
2009
2011
2013
2015
2017
5.1 10.2 MariaDB8.0 MySQL
8.4 PostgreSQL3.25.0 SQLite
7.0 DB2 LUW8i Oracle
2005[0] 2012 SQL Server[0]No framing
OVER (SQL:2003) Availability
1999
2001
2003
2005
2007
2009
2011
2013
2015
2017
5.1 10.2 MariaDB8.0 MySQL
8.4 PostgreSQL3.25.0 SQLite
7.0 DB2 LUW8i Oracle
2005[0] 2012 SQL Server[0]No framing
OVER (SQL:2003) AvailabilityImpalaSpark
NuoDB
BigQuery Hive
OVERSQL:2003 frame exclusion
Since SQL:2003OVER(ORDERBY…BETWEEN…exclude[noothers|currentrow|group|ties])
OVER (frame exclusion)
Since SQL:2003OVER(ORDERBY…BETWEEN…exclude[noothers|currentrow|group|ties])
default
OVER (frame exclusion)
noothers
Since SQL:2003
x12223
OVER(ORDERBY…BETWEEN…exclude[noothers|currentrow|group|ties])
default
OVER (frame exclusion)
noothers
currentrow
Since SQL:2003
x12223
OVER(ORDERBY…BETWEEN…exclude[noothers|currentrow|group|ties])
default
OVER (frame exclusion)
currentrow
groupx=current_row
currentrow
Since SQL:2003
x12223
OVER(ORDERBY…BETWEEN…exclude[noothers|currentrow|group|ties])
default
OVER (frame exclusion)
group
tiesgroupx=current_row
currentrow
Since SQL:2003
x12223
OVER(ORDERBY…BETWEEN…exclude[noothers|currentrow|group|ties])
default
OVER (frame exclusion)
ties
OVER (frame exclusion) Since SQL:20111999
2001
2003
2005
2007
2009
2011
2013
2015
2017
5.1 MariaDBMySQL
11 PostgreSQL3.28.0 SQLite
DB2 LUWOracleSQL Server
SQL:2011
OVERSQL:2011 groups option
OVER (groups option) Since SQL:2011ORDERBYx<frameunit>between1precedingand1following
OVER (groups option) Since SQL:2011ORDERBYx<frameunit>between1precedingand1following
rows,range
OVER (groups option) Since SQL:2011ORDERBYx<frameunit>between1precedingand1following
rows,range
x13
3.53.54
CURRENT ROW
OVER (groups option) Since SQL:2011ORDERBYx<frameunit>between1precedingand1following
rows,range
rowscount(*)
x13
3.53.54
CURRENT ROW
OVER (groups option) Since SQL:2011ORDERBYx<frameunit>between1precedingand1following
rows,range
rangexbetweencurrent_row-1
andcurrent_row+1
rowscount(*)
x13
3.53.54
CURRENT ROW
OVER (groups option) Since SQL:2011ORDERBYx<frameunit>between1precedingand1following
rows,rangeNew in
SQL:2011 groups
rangexbetweencurrent_row-1
andcurrent_row+1
rowscount(*)
x13
3.53.54
CURRENT ROW
groupscount(distinctx)
OVER (groups option) Since SQL:2011ORDERBYx<frameunit>between1precedingand1following
rows,rangeNew in
SQL:2011 groups
rangexbetweencurrent_row-1
andcurrent_row+1
rowscount(*)
x13
3.53.54
CURRENT ROW
Since SQL:2011OVER (groups option)1999
2001
2003
2005
2007
2009
2011
2013
2015
2017
5.1 MariaDBMySQL
11 PostgreSQL3.28.0 SQLite
DB2 LUWOracleSQL Server
System Versioning (Time Traveling)
INSERTUPDATEDELETE
are DESTRUCTIVE
System Versioning The Problem
System Versioning Since SQL:2011Table can be system versioned, application versioned or both.
CREATETABLEt(...,
System Versioning Since SQL:2011Table can be system versioned, application versioned or both.
CREATETABLEt(...,start_tsTIMESTAMP(9)GENERATEDALWAYSASROWSTART,
System Versioning Since SQL:2011Table can be system versioned, application versioned or both.
CREATETABLEt(...,start_tsTIMESTAMP(9)GENERATEDALWAYSASROWSTART,end_tsTIMESTAMP(9)GENERATEDALWAYSASROWEND,
System Versioning Since SQL:2011Table can be system versioned, application versioned or both.
CREATETABLEt(...,start_tsTIMESTAMP(9)GENERATEDALWAYSASROWSTART,end_tsTIMESTAMP(9)GENERATEDALWAYSASROWEND,
PERIODFORSYSTEM_TIME(start_ts,end_ts)
System Versioning Since SQL:2011Table can be system versioned, application versioned or both.
CREATETABLEt(...,start_tsTIMESTAMP(9)GENERATEDALWAYSASROWSTART,end_tsTIMESTAMP(9)GENERATEDALWAYSASROWEND,
PERIODFORSYSTEM_TIME(start_ts,end_ts))WITHSYSTEMVERSIONING
System Versioning Since SQL:2011Table can be system versioned, application versioned or both.
ID Data start_ts end_ts1 X 10:00:00
UPDATE...SETDATA='Y'...
ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00
DELETE...WHEREID=1
INSERT...(ID,DATA)VALUES(1,'X')
System Versioning Since SQL:2011
ID Data start_ts end_ts1 X 10:00:00
UPDATE...SETDATA='Y'...
ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00
DELETE...WHEREID=1
INSERT...(ID,DATA)VALUES(1,'X')
System Versioning Since SQL:2011
ID Data start_ts end_ts1 X 10:00:00
UPDATE...SETDATA='Y'...
ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00
DELETE...WHEREID=1
INSERT...(ID,DATA)VALUES(1,'X')
System Versioning Since SQL:2011
ID Data start_ts end_ts1 X 10:00:00
UPDATE...SETDATA='Y'...
ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00
DELETE...WHEREID=1
ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00 12:00:00
INSERT...(ID,DATA)VALUES(1,'X')
System Versioning Since SQL:2011
Although multiple versions exist, only the “current” one is visible per default.
After 12:00:00, SELECT*FROMt doesn’t return anything anymore.
ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00 12:00:00
System Versioning Since SQL:2011
ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00 12:00:00
With FOR…ASOF you can query anything you like: SELECT*FROMtFORSYSTEM_TIMEASOFTIMESTAMP'2019-08-1010:30:00'
ID Data start_ts end_ts
1 X 10:00:00 11:00:00
System Versioning Since SQL:2011
System Versioning Since SQL:20111999
2001
2003
2005
2007
2009
2011
2013
2015
2017
5.1 10.3 MariaDBMySQLPostgreSQLSQLite
10.1 DB2 LUW10gR1[0] 11gR1[1] Oracle
2016 SQL Server[0]Short term using Flashback.[1]Flashback Archive. Proprietery syntax.
FORPORTIONOF(UPDATE,DELETE)
CREATETABLEt(...,
Since SQL:2011Table can be system versioned, application versioned or both.
FOR PORTION OF
CREATETABLEt(...,start_tsTIMESTAMP(9),
Since SQL:2011Table can be system versioned, application versioned or both.
FOR PORTION OF
CREATETABLEt(...,start_tsTIMESTAMP(9),end_tsTIMESTAMP(9),
Since SQL:2011Table can be system versioned, application versioned or both.
FOR PORTION OF
CREATETABLEt(...,start_tsTIMESTAMP(9),end_tsTIMESTAMP(9),
PERIODFORapp(start_ts,end_ts))
Since SQL:2011Table can be system versioned, application versioned or both.
FOR PORTION OF
ID Data start_ts end_ts1 X 10:00:00 12:00:00
UPDATEtFORPORTIONOFappFROM'10:30:00'TO'11:30:00'SETDATA='Y'
ID Data start_ts end_ts1 X 10:00:00 10:30:001 Y 10:30:00 11:30:001 X 11:30:00 12:00:00
INSERTt(ID,DATA,start_ts,end_ts)VALUES(1,'X','10:00:00','12:00:00')
Since SQL:2011FOR PORTION OF
ID Data start_ts end_ts1 X 10:00:00 12:00:00
UPDATEtFORPORTIONOFappFROM'10:30:00'TO'11:30:00'SETDATA='Y'
ID Data start_ts end_ts1 X 10:00:00 10:30:001 Y 10:30:00 11:30:001 X 11:30:00 12:00:00
INSERTt(ID,DATA,start_ts,end_ts)VALUES(1,'X','10:00:00','12:00:00')
Since SQL:2011FOR PORTION OF
ID Data start_ts end_ts1 X 10:00:00 12:00:00
UPDATEtFORPORTIONOFappFROM'10:30:00'TO'11:30:00'SETDATA='Y'
ID Data start_ts end_ts1 X 10:00:00 10:30:001 Y 10:30:00 11:30:001 X 11:30:00 12:00:00
INSERTt(ID,DATA,start_ts,end_ts)VALUES(1,'X','10:00:00','12:00:00')
Since SQL:2011FOR PORTION OF
1999
2001
2003
2005
2007
2009
2011
2013
2015
2017
5.1 10.4 MariaDBMySQLPostgreSQLSQLite
10.5 DB2 LUWOracleSQL Server
FORPORTIONOF Since SQL:2011
Since SQL:2011Application Versioning
For a useful application versioning, WITHOUTOVERLAPS
constraints are required.
Apparently this was moved to MariaDB 10.5: https://jira.mariadb.org/browse/MDEV-16978
SQL:2016 (released: 2016-12-15)
JSON_TABLE
JSON Since SQL:2016
[{"id":42,"a1":"foo"},{"id":43,"a1":"bar"}]
JSON Since SQL:2016
id a1
42 foo
43 bar
[{"id":42,"a1":"foo"},{"id":43,"a1":"bar"}]
JSON Since SQL:2016
SELECT*FROMJSON_TABLE(?,'$[*]'COLUMNS(idINTPATH'$.id',a1VARCHAR(…)PATH'$.a1'))r
[{"id":42,"a1":"foo"},{"id":43,"a1":"bar"}]
id a142 foo43 bar
JSON_TABLE Since SQL:2016
SELECT*FROMJSON_TABLE(?,'$[*]'COLUMNS(idINTPATH'$.id',a1VARCHAR(…)PATH'$.a1'))r
[{"id":42,"a1":"foo"},{"id":43,"a1":"bar"}]
id a142 foo43 bar
Bind Parameter
JSON_TABLE Since SQL:2016
SELECT*FROMJSON_TABLE(?,'$[*]'COLUMNS(idINTPATH'$.id',a1VARCHAR(…)PATH'$.a1'))r
[{"id":42,"a1":"foo"},{"id":43,"a1":"bar"}]
id a142 foo43 bar
SQL/JSON Path ‣ Query language to
select elements from a JSON document ‣Defined in the
SQL standardBind
Parameter
JSON_TABLE Since SQL:2016
SELECT*FROMJSON_TABLE(?,'$[*]'COLUMNS(idINTPATH'$.id',a1VARCHAR(…)PATH'$.a1'))r
[{"id":42,"a1":"foo"},{"id":43,"a1":"bar"}]
id a142 foo43 bar
SQL/JSON Path ‣ Query language to
select elements from a JSON document ‣Defined in the
SQL standardBind
Parameter
JSON_TABLE Since SQL:2016
SELECT*FROMJSON_TABLE(?,'$[*]'COLUMNS(idINTPATH'$.id',a1VARCHAR(…)PATH'$.a1'))r
[{"id":42,"a1":"foo"},{"id":43,"a1":"bar"}]
id a142 foo43 bar
SQL/JSON Path ‣ Query language to
select elements from a JSON document ‣Defined in the
SQL standardBind
Parameter
JSON_TABLE Since SQL:2016
SELECT*FROMJSON_TABLE(?,'$[*]'COLUMNS(idINTPATH'$.id',a1VARCHAR(…)PATH'$.a1'))r
JSON_TABLE Since SQL:2016
SELECT*FROMJSON_TABLE(?,'$[*]'COLUMNS(idINTPATH'$.id',a1VARCHAR(…)PATH'$.a1'))r
INSERTINTOtarget_table
JSON_TABLE Since SQL:2016
JSON_TABLE Availability1999
2001
2003
2005
2007
2009
2011
2013
2015
2017
MariaDB8.0 MySQL
PostgreSQLSQLite
11.1.4.4 DB2 LUW12cR1 Oracle
2016[0] SQL Server[0]OPENJSON provides similar functionality
PostrgreSQL 12 (beta)
New Standard SQL features in PostgreSQL 12 (likely)
SQL/JSON path language Like XPath for XML and CSS selectors for HTML(the expressive power is somewhere in the middle)
Generated columnsGENERATEDALWAYSAS(<expr>)STORED
Chained Transactions[COMMIT|ROLLBACK]…ANDCHAIN
Hyperbolic functions sinh,cosh,tanh,asinh,acosh,atanh
https://www.flickr.com/photos/mfoubister/25367243054/
https://www.flickr.com/photos/mfoubister/25367243054/
A lot has happened
since SQL-92
https://www.flickr.com/photos/mfoubister/25367243054/
SQL has evolved
beyond the relational idea
A lot has happened
since SQL-92
https://www.flickr.com/photos/mfoubister/25367243054/
SQL has evolved
beyond the relational idea
If you use SQL for
CRUD operations only, you are doing it wrong
A lot has happened
since SQL-92
https://www.flickr.com/photos/mfoubister/25367243054/
SQL has evolved
beyond the relational idea
If you use SQL for
CRUD operations only, you are doing it wrong
A lot has happened
since SQL-92
https://modern-sql.com@ModernSQL by @MarkusWinand
https://www.flickr.com/photos/mfoubister/25367243054/
SQL has evolved
beyond the relational idea
If you use SQL for
CRUD operations only, you are doing it wrong
A lot has happened
since SQL-92
https://modern-sql.com@ModernSQL by @MarkusWinand
Training:https://winand.at/