© 2013 EDB All rights reserved. 1

Exciting Features In PostgreSQL 9.5

• Amit Kapila | 2015.04.11

2

● June 10, 2014 – branch 9.4

● June 2014 – CF1 - Completed

● August 2014 – CF2 - Completed

● October 2014 – CF3 - Completed

● December 2014 – CF4 - Completed

● February 2015 – CF5 – In Progress

Development Status

3

● Developer and SQL Features

● DBA and Administration

● Replication

● Performance

New Features

4

Multi-column subselectUpdate

● Update more than one column with subselect

● SQL standard syntax

UPDATE tab SET (col1, col2) =

(SELECT foo, bar FROM tab2)

WHERE ...

5

SKIP LOCKED● Like SELECT NOWAIT

● Except skip rows instead of error

postgres=# SELECT * FROM a FOR UPDATE NOWAIT;

ERROR: could not obtain lock on row in relation "a"

postgres=# SELECT * FROM a FOR UPDATE SKIP LOCKED;

a | b | c

----+----+----

2 | 2 | 2

3 | 3 | 3

6

SKIP LOCKED● Generally used for Advanced Queuing in databases. To

Dequeue the data from a queue

7

Row Level Security● Allows controlling at row level which rows can be retrieved by SELECT or manipulated using INSERT | UPDATE | DELETE

● Need to define policies for tables using Policy commands (CREATE | ALTER |DROP Policy)

● Row Security needs to be enabled and disabled by the owner on a per-table basis using

ALTER TABLE .. ENABLE/DISABLE ROW SECURITY.

8

Row Level Security● ROW SECURITY is disabled on tables by default and must be enabled for policies on the table to be used.

● If nopolicies exist on a table with ROW SECURITY enabled, a default-deny policy is used and no records will be visible.

● A new role capability, BYPASSRLS, which can only be set by the superuser, is added to allow other users to be able to bypass row security using row_security = OFF

9

Row Level Security● row_security - a new parameter in postgresql.conf controls if row security policies are to be applied to queries which are run against tables that have row security enabled.

on - all users, except superusers and the owner of the table, will have the row policies for the table applied to

their queries.

force - this is to apply policies for superusers and owner of table.

off -will bypass row policies for the table, if user doing operation has BYPASSRLS attribute, and error if not.

10

Row Level Security – How it works● create table clients ( id serial primary key,

account_name text not null unique,

account_manager text not null

);

CREATE TABLE

create user peter;

CREATE ROLE

create user joanna;

CREATE ROLE

create user bill;

CREATE ROLE

11

Row Level Security – How it works● Grant appropriate permissions

grant all on table clients to peter, joanna, bill;

GRANT

grant all on sequence clients_id_seq to peter, joanna, bill;

GRANT

● Populate the table

insert into clients (account_name, account_manager)

values ('initrode', 'peter'), ('initech', 'bill'), ('chotchkie''s', 'joanna');

INSERT 0 3

12

Row Level Security – How it works● By default, all the rows are visible.

$ \c - peter

$ select * from clients;

id | account_name | account_manager

----+--------------+-----------------

1 | initrode | peter

2 | initech | bill

3 | chotchkie's | joanna

(3 rows)

13

Row Level Security – How it works● Now lets create policies and enable row level security

create policy just_own_clients on clients

for all

to public

using ( account_manager = current_user );

CREATE POLICY

alter table clients ENABLE ROW LEVEL SECURITY;

ALTER TABLE

14

Row Level Security – How it works● Now, I can only see rows belonging to myself:

$ select * from clients;

id | account_name | account_manager

----+--------------+-----------------

1 | initrode | peter

(1 row)

$ \c - joanna

$ select * from clients;

id | account_name | account_manager

----+--------------+-----------------

3 | chotchkie's | joanna

(1 row)

15

● Developer and SQL Features

● DBA and Administration

● Replication

● Performance

New Features

16

min and max wal size● checkpoint_segments removed!● Instead, control min and max size● min_wal_size (default 80MB)● max_wal_size (default 1GB)● Checkpoints auto-tuned to happen in between● Moving average of previous checkpoints● Space only consumed when actually needed

17

Foreign Table Inheritance● Foreign tables can now be inheritance children, or parents.

● PostgreSQL offers a way to do partitioning by using

table inheritance and CHECK constraints

● This feature can be used for sharding

18

Commit Timestamp Tracking● Optional tracking of commit timestamps

● track_commit_timestamp=on

● Default is off and changing the value of this parameter requires server restart

● User can retrieve the information for transactions that were committed after above option is enabled

● Can be used by multimaster systems for conflict resolution

19

Commit Timestamp Tracking● postgres=# SELECT xmin, pg_xact_commit_timestamp(xmin) FROM a;

xmin | pg_xact_commit_timestamp

------+-------------------------------

787 | 2015-03-15 15:09:52.253007+00

● postgres=# SELECT * FROM pg_last_committed_xact();

xid | timestamp

-----+-------------------------------

791 | 2015-03-15 15:11:38.709125+00

20

● Developer and SQL Features

● DBA and Administration

● Replication

● Performance

New Features

21

pg_rewind● a tool for synchronizing a PostgreSQL cluster with another copy of the same cluster, after the clusters' timelines have diverged

● This is used to bring an old master server back online after failover, as a standby that follows the new master

● The advantage of pg_rewind over taking a new base backup, or tools like rsync, is that pg_rewind does not require reading through all unchanged files in the cluster

22

pg_rewind● It is lot faster when the database is large and only a small portion of it differs between the clusters

● The target server (old-master) must be shut down cleanly before running pg_rewind

● pg_rewind requires that the wal_log_hints option is enabled in postgresql.conf, or that data checksums were enabled when the cluster was initialized with initdb.

● full_page_writes must also be enabled.

23

● Developer and SQL Features

● DBA and Administration

● Replication

● Performance

New Features

24

BRIN● Block Range Index

● Stores only bounds-per-block-range

● Default is 128 blocks

● Very small indexes

● Scans all blocks for matches

● Used for scanning large tables

25

BRIN

=# CREATE TABLE brin_example AS SELECT generate_series(1,100000000) AS id;

SELECT 100000000

=# CREATE INDEX btree_index ON brin_example(id);

CREATE INDEX

Time: 239033.974 ms

=# CREATE INDEX brin_index ON brin_example USING brin(id);

CREATE INDEX

Time: 42538.188 ms

26

BRIN=# CREATE TABLE brin_example AS SELECT generate_series(1,100000000) AS id;

SELECT 100000000

=# CREATE INDEX btree_index ON brin_example(id);

CREATE INDEX

Time: 239033.974 ms

=# CREATE INDEX brin_index ON brin_example USING brin(id);

CREATE INDEX

Time: 42538.188 ms

Conclusion – Brin index creation is much faster

27

BRIN – Index creation with different block ranges

=# CREATE INDEX brin_index_64 ON brin_example USING brin(id) WITH (pages_per_range = 64);

CREATE INDEX

=# CREATE INDEX brin_index_256 ON brin_example USING brin(id) WITH (pages_per_range = 256);

CREATE INDEX

=# CREATE INDEX brin_index_512 ON brin_example USING brin(id) WITH (pages_per_range = 512);

CREATE INDEX

28

BRIN – Size relname | pg_size_pretty

----------------+----------------

brin_example | 3457 MB

brin_index | 104 kB

brin_index_256 | 64 kB

brin_index_512 | 40 kB

brin_index_64 | 192 kB

btree_index | 2142 MB

(6 rows)

Conclusion – Brin indexes are smaller in size

29

BRIN – How it works● A new index access method intended to accelerate scans of very large tables, without the maintenance overhead of btrees or other traditional indexes.

● They work by maintaining "summary" data about block ranges.

30

BRIN – How it works● For data types with natural 1-D sort orders like integers, the summary info consists of the maximum and the minimum values of each indexed column within each page range

● As new tuples are added into the index, the summary information is updated if the block range in which the tuple is added is already summarized

● Otherwise subsequent pass of Vacuum or the brin_summarize_new_values() function will create the summary information.

31

Read Scalability● We will see a boost in scalability for read workload when the data can fit in RAM.  I have ran a pgbench read-only load to compare the performance difference between 9.4 and HEAD (62f5e447)on IBM POWER-8 having 24 cores, 192 hardware threads, 492GB RAM

● The data is mainly taken for 2 kind of workloads, when all the data fits in shared buffers (scale_factor = 300) and when all the data can't fit in shared buffers, but can fit in RAM (scale_factor = 1000)

32

Read Scalability – Data fits in shared_buffers

1 8 16 32 64 128 2560

100000

200000

300000

400000

500000

600000

pgbench -S -M prepared, PG9.5dev as of commit 62f5e4

median of 3 5-minute runs, scale_factor = 300, max_connections = 300, shared_buffers = 8GB

9.4

HEAD

Client Count

TP

S

33

Read Scalability● In 9.4 it peaks at 32 clients, now it peaks at 64 clients and we can see the performance improvement upto (~98%) and it is better in all cases at higher client count starting from 32 clients

● The main work which lead to this improvement is commit – ab5194e6 (Improve LWLock scalability)

34

Read Scalability● The previous implementation has a bottleneck around spin locks that were acquired for LWLock Acquisition and Release and the implementation for 9.5 has changed the LWLock implementation to use atomic operations to manipulate the state.

35

Read Scalability – Data fits in RAM

1 8 16 32 64 128 2560

50000

100000

150000

200000

250000

300000

350000

400000

pgbench -S -M prepared, PG9.5dev as of commit 62f5e4

median of 3 5-minute runs, scale_factor = 1000, max_connections = 300, shared_buffers = 8GB

9.4

HEAD

Client Count

TP

S

36

Read Scalability● In this case, we could see the good performance improvement (~25%)even at 32 clients and it went upto (~96%) at higher client count, in this case also where in 9.4 it was peaking at 32 client count, now it peaks at 64 client count and the performance is better atall higher client counts.

● The main work which lead to this improvement is commit – ab5194e6 (Improve LWLock scalability)

37

Read Scalability● In this case there were mainly 2 bottlenecks

● a BufFreeList LWLock was getting acquired to find a free buffer for a page

● to change the association of buffer in buffer mapping hash table a LWLock is acquired on a hash partition towhich the buffer to be associated belongs and as there were just 16 such partitions

38

Read Scalability● To reduce the bottleneck due to first problem, used a spinlock which is held just long enough to pop the freelist or advance the clock sweep hand, and then released

● To reduce the bottleneck due to second problem, increase the buffer partitions to 128

● The crux of this improvement is that we had to resolve both the bottlenecks together to see a major improvement in scalability

39

Parallel Vacuumdb● vacuumdb can use concurrent connections

● Add -j<n> to command line

● Speed up important VACUUM or ANALYZE

● This option reduces the time of the processing but it also increases the load on the database server.This option reduces the time of the processing but it also increases the load on the database server.

40

Sorting Improvements● Use abbreviated keys for faster sortingof text

● transformation of strings into binary keys using strxfrm(), and sorting the keys instead

● using a strcmp()-based comparator with the keys, which only considers raw byte ordering

● abbreviate by taking the first few characters of the strxfrm() blob.

41

Sorting Improvements● If the abbreviated comparison is insufficent to resolve the comparison, we fall back on the normal comparator.

● This can be much faster than the old way of doing sorting if the first few bytes of the string are usually sufficient to resolve the comparison.

42

Sorting Improvements● As an example

create table stuff as select random()::text as a, 'filler filler filler'::text as b, g as c from generate_series(1, 1000000) g;

SELECT 1000000

create index on stuff (a);

CREATE INDEX

● On PPC64 m/c, before this feature, above operation use to take 6.3 seconds and after feature it took just 1.9 seconds, which is 3x improvement. Hooray!

43

WAL Compression● Optional compression for full page images in WAL

● wal_compression=on

● Default is off, can be set by user and doesn't require restart

● Support for compressing full page images

44

WAL Compression● Smaller WAL

● Faster writes, faster replication

● Costs CPU

● Only compresses FPIs

● Still useful to gzip archives!

45

Index Scan Optimization● improved performance for Index Scan on ">" condition

● We can see performance improvement from 5 to 30 percent.

46

● Thanks to Magnus Hagander who has presented the paper for PostgreSQL 9.5 features in PGConf US 2015. Some of the slides in this paper are from his paper. You can download his slides from http://www.hagander.net/talks/

● Thanks to Hubert 'depesz' Lubaczewski and Michael Paquier for writing blogs for new features in PostgreSQL 9.5. Some of the examples used in this paper are taken from their blogs.

47

Questions?

48

Thanks!