47
TODAY
1. Interacting with a DMBS
2. Non trivial features of PG
3. DEMOs
1
THE PSQL UTILITY
Command
p s q l −h <s e r v e r > −U <us e r> <database>
Remote server
p s q l −h s c i −d i d a t t i c a . u n i t n . i t −U db 001 db 001
Local server
p s q l db 001
2
INSTALL
Client only
• Debian/Ubuntu: postgresql-client
• Brew: libpq
Server & Client
• Debian: postgresql
• Brew: postgresql
• macOS: Potgres.app
GUI
• web app: pgAdmin 4
3
VPN
The server is reachable only from the local network
unitn, unitn-x, (local) eduroam, cabled
wiki.unitn.it/pub:conf-vpn-paloalto-en
wiki.unitn.it/pub:conf-vpn
4
ACCOUNTS
gspreadsheets: 1IrRufeuw12lymAtuWWRgH1g0NUlqhzb1ClpEmaSPwcs
Structure
• Database: db ddd
• Username: db ddd
• Password: pass ddd
Where ddd in [001, 500].
Change your password
# Connect v i a p s q l
PGPASSWORD=p a s s 0 0 1 p s q l \−h s c i −d i d a t t i c a . u n i t n . i t −U db 001 db 001
# Change password v i a query
ALTER USER ” db 001 ” WITH PASSWORD ’ newpass ’ ;
5
FAQ 1/2
Quotation marks
"" : names of schemas, tables, columns, etc.
’’ : string constant.
Cast
−− SQL s tanda rd
CAST ( e x p r e s s i o n AS t y p e )
−− Old s t y l e PG
e x p r e s s i o n : : t y p e
Auto increment
SERIAL – see documentation.
6
FAQ 2/2
Case Sensitivity
Tables, columns names: yes with ""
. . . without, transformed to lowercase.
String: case sensitive.
Naming things - doc.
Database Cluster (Server - User & Groups)
→ Databases (Name Space) → Schemas → Tables.
7
JDBC & psycopg2
JDBC - CONNECT
/∗ JDBCExample . j a v a ∗/import j a v a . s q l . ∗ ;
/∗ c l a s s JDBCExample > s t a t i c vo i d main ( ) ∗/S t r i n g u r l = ” jdbc : p o s t g r e s q l : // s c i −d i d a t t i c a . un i t n . i t / db 001 ” ;
P r o p e r t i e s p rops = new P r o p e r t i e s ( ) ;
p rops . s e tP r o p e r t y ( ” u s e r ” , ” db 001 ” ) ;
p rops . s e tP r o p e r t y ( ” password ” , ” s e c r e t 0 0 1 ” ) ;
p rops . s e tP r o p e r t y ( ” s s l ” , ” f a l s e ” ) ;
Connect ion conn = Dr iverManager . g e tConnec t i on ( u r l , p rops ) ;
/∗ Do s t u f f . ∗/conn . c l o s e ( ) ;
# Download jdbc . p o s t g r e s q l . o rg /download/ po s t g r e s q l −42 .2 . 8 . j a r
j a v a −cp ”$ (pwd)/ p o s t g r e s q l −42 .2 . 8 . j a r ; $ (pwd)/ ” JDBCExample
https://jdbc.postgresql.org/documentation/head
8
JDBC - QUERY
Statement s t = conn . c r e a t eS ta t emen t ( ) ;
R e s u l t S e t r s = s t . executeQuery (
”SELECT ∗ FROM mytab le WHERE foo = 500” ) ;
w h i l e ( r s . nex t ( ) ) {System . out . p r i n t ( ”Column 1 r e t u r n e d ” ) ;
System . out . p r i n t l n ( r s . g e t S t r i n g ( 1 ) ) ;
}r s . c l o s e ( ) ;
s t . c l o s e ( ) ;
https://jdbc.postgresql.org/documentation/head/query.html
9
JDBC - PREPARED STATEMENT
S t r i n g q = ”INSERT INTO t b l ( foo ) VALUES (? ) ” ;
PreparedStatement s t = conn . p r epa r eS ta t ement ( q ) ;
s t . s e tOb j e c t (1 , new va lue ) ;
s t . executeUpdate ( ) ;
s t . c l o s e ( ) ;
jdbc.postgresql.org/documentation/head/update.html
jdbc.postgresql.org/documentation/head/java8-date-time.html
10
PYTHON
import psycopg2 # i n i t d . org / psycopg /
# Connect
conn = psycopg2 . connect ( ”dbname=’db 001 ’ u s e r =’db 001 ’ ” +
” hos t=’ s c i −d i d a t t i c a . un i t n . i t ’ password=’ s e c r e t 0 0 1 ’ ” )
cu r = conn . c u r s o r ( )
# Create / Prepared s ta tement
cu r . e x e cu t e ( ”CREATE TABLE t e s t . . . ” )
cu r . e x e cu t e ( ”INSERT INTO t e s t (num , data ) VALUES (%s , %s ) ” ,
(100 , ” abc ’ d e f ” ) )
conn . commit ( ) # Make changes permanent
# Query
cu r . e x e cu t e ( ”SELECT ∗ FROM t e s t ; ” )
cu r . f e t c hone ( ) # next row on l y
cu r . f e t c h a l l ( ) # a l l r ema in i ng ones
# Clean up
cu r . c l o s e ( )
conn . c l o s e ( )
https://wiki.postgresql.org/wiki/Psycopg2_Tutorial
11
Nice types
RANGE
Types
numrange, tstzrange, daterange
Syntax’ ( 7 , 1 2 ] ’ : : numrange
numrange ( 7 , 12 , ’ ( ] ’ )
Operators
www.postgresql.org/docs/current/functions-range.html#
RANGE-OPERATORS-TABLE
@>, &&, >> &>, −|−, +, ∗ , −
Functions
www.postgresql.org/docs/current/functions-range.html#
RANGE-FUNCTIONS-TABLE
lower, upper, isempty, lower inc, lower inf, range merge.
12
RANGE
Types
numrange, tstzrange, daterange
Syntax’ ( 7 , 1 2 ] ’ : : numrange
numrange ( 7 , 12 , ’ ( ] ’ )
Operators
www.postgresql.org/docs/current/functions-range.html#
RANGE-OPERATORS-TABLE
@>, &&, >> &>, −|−, +, ∗ , −
Functions
www.postgresql.org/docs/current/functions-range.html#
RANGE-FUNCTIONS-TABLE
lower, upper, isempty, lower inc, lower inf, range merge.
12
RANGE
Types
numrange, tstzrange, daterange
Syntax’ ( 7 , 1 2 ] ’ : : numrange
numrange ( 7 , 12 , ’ ( ] ’ )
Operators
www.postgresql.org/docs/current/functions-range.html#
RANGE-OPERATORS-TABLE
@>, &&, >> &>, −|−, +, ∗ , −
Functions
www.postgresql.org/docs/current/functions-range.html#
RANGE-FUNCTIONS-TABLE
lower, upper, isempty, lower inc, lower inf, range merge.12
ARRAYS
Declaration
f i e l d 1 t y p e [ ] ,
f i e l d 2 t y p e [ ] [ ] ,
f i e l d 3 t y p e [ n ] [m] [ k ] ,
. . .
Syntax
’ {1 , 2 , 3 , 4} ’
’ {{1 , 2 , 3} ,{4 , 5 , 6} ,{7 , 8 , 9}} ’
ARRAY[ 1 , 2 , 3 , 4 ]
Navigation
f i e l d 1 [ 0 ] , f i e l d 2 [ 2 : 4 ] [ : 3 ]
13
ARRAYS
Editing
f i e l d 1 [ 0 ] = 33
f i e l d 2 [ 1 ] [ : 3 ] = ’{2 , 3 , 4} ’
a r r 1 | | a r r 2
a r r 1 | | v a l | | NULL
a r r a y c a t ( )
Search
ANY, ALL , &&,
a r r a y p o s i t i o n , a r r a y p o s i t i o n s
www.postgresql.org/docs/current/arrays.html
14
JSON
JSON & JSONb
Navigation
−> # descend one node , r e t u r n s JSON node
−>> # descend one node , r e t u r n s va lue /JSON as t e x t
#> # q u e r y path , r e t u r n s JSON node
#>> # q u e r y path , r e t u r n s va lue /JSON as t e x t
15
JSON
JSON & JSONb
Filter / Modify JSONb only.
Containment : <@, ? , ? |Concat : | |Delete : −, #−Path q u e r y (& compare ) : @? , @@
16
HSTORE
Text only, dictionary.
Mostly replaced by JSONb.
17
Cool operators and functions
WINDOW FUNCTIONS
Like aggregation, but does not collapse rows.
f u n c ( ) [ FILTER ( WHERE f i l t e r c l a u s e ) ] OVER ( window def )
Additional functions 1
www.postgresql.org/docs/12/functions-window.html
• row number()
• lag, lead (expression, offset, default)
• first value, last value, (expression)
• nth value (expression, nth)
18
WINDOW FUNCTIONS
Like aggregation, but does not collapse rows.
f u n c ( ) [ FILTER ( WHERE f i l t e r c l a u s e ) ] OVER ( window def )
Window definition 2
• PARTITION BY
• ORDER BY
• frame clause∗
www.postgresql.org/docs/12/sql-expressions.html#
SYNTAX-WINDOW-FUNCTIONS
19
WINDOW FUNCTIONS
Like aggregation, but does not collapse rows.
f u n c ( ) [ FILTER ( WHERE f i l t e r c l a u s e ) ] OVER ( window def )
Usage examples
“First n items for each group”, “running sums”, “stats”, . . .
Buy at least 4 and the 2 less expensive are free.
SELECT c u s t o m e r i d , SUM( p r i c e ) FROM (
SELECT c u s t o m e r i d , p r i c e ,
ROW NUMBER( ) as r
OVER (PARTITION BY c u s t o m e r i d ORDER BY p r i c e ) ,
COUNT( ) as m OVER (PARTITION BY c u s t o m e r i d ) ,
) tmp WHERE m < 4 OR (m >= 4 AND r > 2)
20
SET RETURNING FUNCTIONS
Series Generating Functions
g e n e r a t e s e r i e s ( s t a r t , stop , s t e p )
g e n e r a t e s u b s c r i p t s ( a r r a y , dim , r e v e r s e )
www.postgresql.org/docs/current/functions-srf.html
21
Indexes
INDEXES OVERVIEW
• B-tree (default)
• Hash
• GiST
• SP-GiST
• GIN
• BRIN
www.postgresql.org/docs/current/indexes-types.html
22
B-TREE
Speeds up
• <,<=,=, >=, >
→ BETWEEN, IN
• LIKE
constant pattern anchored to the beginning, e.g. ’foo%’
• IS NULL, IS NOT NULL
23
HASH
Speeds up
• =
24
GENERALIZED SEARCH TREE (GIST)
Balanced, tree-structured access method.
Template in which to implement arbitrary indexing schemes.
Natively supports
• Geometry: box, circle, point, polygon
• Internet Addresses: inet, cidr
• any range type
• tsquery, tsvector
document representation to optimized full text search
Many more from the contrib collection.
www.postgresql.org/docs/current/gist-intro.html
25
SP-GIST
Similar to GiST but support non-balanced disk-based data
structures.
e.g. quadtrees, k-d trees, and radix trees (tries)
26
GIN
Inverted indexes used for complex objects.
Natively supports
• array
• jsonb
• tsvector
www.postgresql.org/docs/current/gin-builtin-opclasses.html
27
BLOCK RANGE INDEXES (BRIN)
Used for very large tables.
Indexes range of consecutive table’s physical blocks.
www.postgresql.org/docs/current/brin-intro.html
28
Explain!
THE EXPLAIN COMMAND
“show the execution plan of a statement”
For each query node shows
e.g. (cost=0.00..0.24 rows=10 width=82)
• cost=f..a
f: Cost up to the first row
a: Cost for computing all rows
• rows
Expected number of returned rows
• width
Expected width of the rows
www.postgresql.org/docs/current/sql-explain.html
29
EXPLAIN ANALYZE
Executes the command and
shows actual run times and other statistics.
For each query node shows
e.g. . . . (actual time=0.025..0.029 rows=10 loops=1)
• actual time
Average execution time for one loop
• rows
Actual number of returned rows
• loops
Times the node has been executed
30
EXPLAIN ANALYZE EXAMPLE 1
EXPLAIN ANALYZE
SELECT surname , name
FROM p e o p l e p e r s o n LIMIT 1 0 ;
L i m i t ( c o s t = 0 . 0 0 . . 0 . 2 4 rows=10 width =15)
( a c t u a l t ime = 0 . 0 1 6 . . 0 . 0 2 2 rows=10 l o o p s =1)
−> Seq Scan on p e o p l e p e r s o n
( c o s t = 0 . 0 0 . . 5 8 8 . 8 8 rows =24988 width =15)
( a c t u a l t ime = 0 . 0 1 4 . . 0 . 0 1 7 rows=10 l o o p s =1)
P l a n n i n g t ime : 0 . 1 4 1 ms
E x e c u t i o n t ime : 0 . 0 4 3 ms
31
EXPLAIN ANALYZE EXAMPLE 1
EXPLAIN ANALYZE
SELECT surname , name
FROM p e o p l e p e r s o n LIMIT 1 0 ;
L i m i t ( c o s t = 0 . 0 0 . . 0 . 2 4 rows=10 width =15)
( a c t u a l t ime = 0 . 0 1 6 . . 0 . 0 2 2 rows=10 l o o p s =1)
−> Seq Scan on p e o p l e p e r s o n
( c o s t = 0 . 0 0 . . 5 8 8 . 8 8 rows =24988 width =15)
( a c t u a l t ime = 0 . 0 1 4 . . 0 . 0 1 7 rows=10 l o o p s =1)
P l a n n i n g t ime : 0 . 1 4 1 ms
E x e c u t i o n t ime : 0 . 0 4 3 ms
31
EXPLAIN ANALYZE EXAMPLE 2 - QUERY
EXPLAIN ANALYZE
SELECT p . surname , p . name , c . name ,
EXTRACT(YEAR FROM AGE( b i r t h d a t e ) ) ,
FROM p e o p l e p e r s o n p JOIN p e o p l e c o u n t r y c
ON c i t i z e n s h i p i d=c . i d
WHERE EXTRACT(YEAR FROM AGE( b i r t h d a t e ) ) > 2 5 ;
32
EXPLAIN ANALYZE EXAMPLE 2 - RESULT
Hash J o i n ( c o s t = 8 . 4 6 . . 1 0 9 8 . 4 3 rows =8329 width =34)
( a c t u a l t ime = 0 . 2 9 4 . . 6 9 . 5 4 5 rows =14518 l o o p s =1)
Hash Cond : ( p . c i t i z e n s h i p i d = c . i d )
−> Seq Scan on p e o p l e p e r s o n p
( c o s t = 0 . 0 0 . . 9 6 3 . 7 0 rows =8329 width =23)
( a c t u a l t ime = 0 . 0 1 9 . . 3 9 . 5 3 5 rows =14596 l o o p s =1)
F i l t e r : ( d a t e p a r t ( ’ year ’ : : t e x t , age ( (CURRENT DATE ) : : t imestamp w i t h t ime zone , ( b i r t h d a t e ) : : t imestamp w i t h t ime zone ) ) > ’ 2 5 ’ : : d o u b l e p r e c i s i o n )
Rows Removed by F i l t e r : 10392
−> Hash ( c o s t = 4 . 8 7 . . 4 . 8 7 rows =287 width =15)
( a c t u a l t ime = 0 . 2 5 0 . . 0 . 2 5 1 rows =287 l o o p s =1)
Buckets : 1024 Batches : 1 Memory Usage : 22kB
−> Seq Scan on p e o p l e c o u n t r y c
( c o s t = 0 . 0 0 . . 4 . 8 7 rows =287 width =15)
( a c t u a l t ime = 0 . 0 0 8 . . 0 . 1 2 5 rows =287 l o o p s =1)
P l a n n i n g t ime : 0 . 5 5 0 ms
E x e c u t i o n t ime : 70 .285 ms
33
Never ending
• Foreign data wrappers
www.postgresql.org/docs/current/ddl-foreign-data.html
• PostGIS
postgis.net
• Postgres Weekly
postgresweekly.com
• . . .
34
DEMO
Q&A
