A Peek in the Elephant's Trunk

© 2014 EnterpriseDB Corporation. 1

A Peek in the Elephants Trunk Dave Page | 2014-04-04


•  PostgreSQL −  Core Team Member −  pgAdmin Project Lead −  Secretary, PostgreSQL Europe −  Vice-chair, PostgreSQL Community Association of Canada

•  EnterpriseDB −  Chief Architect, Tools & Installers −  Tea boy

Dave Page


•  Discuss the idea on pgsql-hackers −  Ensure no-one else is already doing the same work −  Gain acceptance for the idea −  Get consensus for the design and implementation plan

•  Post the patch to the mailing list −  Add it to commitfest.postgresql.org −  Discuss and implement improvements until everyone is happy −  Reviews and feedback generally come during the commitfest

•  Breathe a sigh of relief as your work is committed −  Profit! (when the next major release is fully baked)

Patch lifecycle


•  Most developers re-focus on reviewing and committing

•  Commitfests happen four times per year −  June, September, November, January −  Commitfests usually last a month −  Except January, which is normally longer

Commitfests


(as of 1st April)

Statistics

•  4 commitfests closed •  169 patches committed •  39 patches rejected

•  24 patches awaiting review •  15 patches ready for a committer •  11 patches returned with feedback or waiting on the

author •  341 patches in total (though some appear in multiple

commitfests)!


Hackers – feel free to chime in with additional details, comments or corrections!!

So what’s new in 9.4?


Performance


•  XLogInsert scaling −  Improves the scaling of WAL insert insert operations by

reducing lock contention. This allows for greater write scaling.

−  Author: Heikki Linnakangas −  Reviewers: Andres Freund, Abhijit Menon-Sen −  Committer: Heikki Linnakangas

Performance


•  Reduce WAL for update operations −  Reduce the size of WAL records generated for UPDATE

operations, leading to less WAL data, reduced I/O etc.

−  Authors: Amit Kapila, Hari Babu −  Reviewers: Robert Haas, Heikki Linnakangas −  Committer: Heikki Linnakangas

Performance


•  pg_prewarm −  A contrib module that provides a pg_prewarm() SQL function

that can be used to load parts of relations into the OS cache (prefetch or a sync read) or the PostgreSQL buffer cache. The user can specify the relation, fork, and optionally start & end blocks to load.

−  Author: Robert Haas −  Reviewers: Álvaro Herrera, Amit Kapila, Jeff Janes, Gurjeet

Singh, and others −  Committer: Robert Haas

Performance


psql (9.4devel) Type "help" for help. -- Install the extension pgbench=# CREATE EXTENSION pg_prewarm; CREATE EXTENSION -- Prewarm the pgbench_accounts table by loading the -- main fork (heap) into shared buffers pgbench=# SELECT pg_prewarm('pgbench_accounts'); pg_prewarm ------------ 163935 (1 row)

pg_prewarm - Example


Server Features


•  Extensible external TOAST tuple support −  Infrastructure to allow future development of alternative

compression schemes for out-of-line TOAST tuples.

−  Author: Andres Freund −  Reviewers: Hitoshi Harada, Robert Haas −  Committer: Robert Haas

Server Features


•  JSONB −  This is a new JSON datatype that stores JSON documents in a

binary format that doesn't need to be re-parsed when accessed. This allows for much faster querying and indexing.

−  Authors: Oleg Bartunov, Teodor Sigaev, Alexander Korotkov, Peter Geoghegan and Andrew Dunstan

−  Reviewer: Andres Freund −  Committer: Andrew Dunstan

Server Features


1 Using jsonb_hash_ops 2 Targeted index on a single key 3 Producing a count based on a search for a particular nested value

https://plus.google.com/+ThomBrownUK/posts/1JizRBGPYBq

JSONB - Thom’s quickie benchmark

JSON JSONB MongoDB

Table Size 1322MB 1375MB 1666MB

Index Size 630MB 283MB1 N/A

Targeted Index Size2 79MB 58MB1 N/A

Speed3 18.759ms 1.111ms 2ms


-- Create a GIN jsonb_hash_ops index CREATE INDEX docs_idx ON docs USING GIN (doc jsonb_hash_ops); -- Containment: Is one document contained within another? postgres=# SELECT postgres-# '{"product": "PostgreSQL", "version": 9.4}'::jsonb postgres-# @> postgres-# '{"version": 9.4}'::jsonb; ?column? ---------- t (1 row)

JSONB - Examples


System Administration


•  ALTER SYSTEM −  Allows the user to edit the server configuration file via SQL.

Changes are written to postgresql.auto.conf, which is read after postgresql.conf and will override settings found there.

−  Author: Amit Kapila −  Reviewer: Hari Babu −  Committer: Tatsuo Ishii



-- Check what work_mem is currently set to postgres=# SHOW work_mem; work_mem ---------- 8MB (1 row) -- Update the system config to set it to a higher value postgres=# ALTER SYSTEM SET work_mem = '16MB'; ALTER SYSTEM

ALTER SYSTEM - Example


-- Reload the server configuration postgres=# SELECT pg_reload_conf(); pg_reload_conf ---------------- t (1 row) -- Et voilà! postgres=# SHOW work_mem; work_mem ---------- 16MB (1 row)



viper:pgsql-dev dpage$ cat data/postgresql.auto.conf # Do not edit this file manually! # It will be overwritten by ALTER SYSTEM command. work_mem = '16MB'



•  ALTER TABLESPACE...MOVE ALL TO... −  Allows bulk moving of relations in tablespaces:

ALTER TABLESPACE name MOVE { ALL | TABLES | INDEXES | MATERIALIZED VIEWS } [ OWNED BY role_name [, ...] ] TO new_tablespace [ NOWAIT ] NOWAIT causes it to bail out if locks cannot be obtained immediately. ALL means all to superusers.

−  Author: Stephen Frost −  Reviewer: Robert Haas −  Committer: Stephen Frost



•  Relocation of tablespaces in pg_base_backup −  Adds a new "-T olddir=newdir" option to pg_basebackup to

allow tablespaces to be relocated easily.

−  Author: Steeve Lennmark −  Reviewer: Peter Eisentraut −  Committer: Peter Eisentraut



•  Base Backup throttling −  Adds a new "--max-rate=N" option to pg_basebackup that

throttles the backup rate to N, where N is 32KB - 1024MB per second. Affects the data directory and "fetched" WAL only. Has no effect on streamed WAL.

−  Authors: Antonin Houska, Álvaro Herrera −  Reviewers: Zoltán Böszörményi, Stefan Radomski, Andres

Freund, Fujii Masao −  Committer: Álvaro Herrera



Monitoring


•  pg_stat_archiver statistics view −  A new view that shows stats about the archiver: number of

WAL files archived, name of the last archived file, time of the last successful archive operation, number of failed attempts, name of the last file that failed, time of the last failure and the time at which the stats were last reset.

−  Author: Gabriele Bartolini −  Reviewers: Vik Fearing, Michael Paquier, Fujii Masao −  Committer: Fujii Masao

Monitoring


•  Include planning time in EXPLAIN ANALYZE −  Simply adds the planning time to the output from EXPLAIN

ANALYZE.

−  Author: Andreas Karlsson −  Reviewer: Robert Haas −  Committer: Robert Haas

Monitoring


•  Improve EXPLAIN to show grouping columns in Agg/Group nodes −  Includes the list of grouping columns (or expressions) in

aggregate and group nodes in EXPLAIN output.

−  Author: Kevin Grittner −  Committer: Tom Lane

Monitoring


-- Get the analysed query plan for a simple aggregate query pgbench=# EXPLAIN ANALYZE SELECT sum(abalance) FROM pgbench_accounts GROUP BY bid; QUERY PLAN --------------------------------------------------------------- HashAggregate (cost=313935.00..313936.00 rows=100 width=8) (actual time=2666.421..2666.436 rows=100 loops=1) Group Key: bid -> Seq Scan on pgbench_accounts (cost=0.00..263935.00 rows=10000000 width=8) (actual time=0.006..646.185 rows=10000000 loops=1) Planning time: 0.061 ms Total runtime: 2666.480 ms (5 rows)

EXPLAIN/EXPLAIN ANALYZE - Example


Replication


•  Recovery target = immediate −  Adds a new recovery target option (immediate) that will cause

recovery to complete as soon as a consistent state is reached. Previously you could recover to the end of the WAL (the default), a named restore point, a timestamp or a transaction ID.

−  Author: MauMau −  Committer: Heikki Linnakangas

Replication


•  Time delayed standbys −  Adds a min_recovery_apply_delay option that can be used in

recovery.conf files on standby servers to delay replay of WAL by the specified amount of time. This can be useful to recover from user errors etc, by halting replication after an error has been spotted, but before replay occurs on the standby.

−  Author: Fabrízio de Royes Mello −  Reviewers: Mitsumasa KONDO, Jaime Casanova −  Committer: Simon Riggs

Replication


•  Logical decoding −  Allows logical changeset data to be extracted from the WAL

stream and consumed via SQL statements or using the streaming replication protocol.

−  Author: Andres Freund −  Reviewers: Kevin Grittner, Robert Haas, Tom Lane, Steve

Singer −  Committer: Robert Haas

Replication


-- Create a replication slot pgbench=# SELECT * FROM pg_create_logical_replication_slot('pgbench_slot', 'test_decoding'); slotname | xlog_position --------------+--------------- pgbench_slot | 0/33D7B550 (1 row) -- Check for replication data pgbench=# SELECT * FROM pg_logical_slot_get_changes('pgbench_slot', NULL, NULL); Location | xid | data ----------+-----+------ (0 rows)

Logical Decoding - Example


-- Do some work pgbench=# INSERT INTO pgbench_branches VALUES (999, 0, NULL); INSERT 0 1 -- Get the replication data pgbench=# SELECT * FROM pg_logical_slot_get_changes('pgbench_slot', NULL, NULL); location | xid | data -----------+-------+------------------------------------------ 0/33D7B550 | 57844 | BEGIN 57844 0/33D7B550 | 57844 | table public.pgbench_branches: INSERT: bid[integer]:999 bbalance[integer]:0 filler[character]:null 0/33D7B620 | 57844 | COMMIT 57844 (3 rows) -- Drop the replication slot pgbench=# SELECT pg_drop_replication_slot('pgbench_slot'); pg_drop_replication_slot -------------------------- (1 row)

Logical Decoding - Example


SQL


•  Refresh materialised views concurrently −  Allows materialised views to be refreshed without affecting

concurrent reads, using transactional semantics.

−  Author: Kevin Grittner −  Reviewer: Hitoshi Harada −  Committer: Kevin Grittner

SQL


•  WITH CHECK option for auto-updateable views −  At present you can insert a record into a view that would

actually be filtered out by the view, but is valid in the underlying table. This adds options to allow views to validate data and reject rows that wouldn't be visible. These checks can be cascaded to underlying views as well.

−  Author: Dean Rasheed −  Reviewer: Pavel Stehule −  Committer: Stephen Frost

SQL


•  UNNEST()/WITH ORDINALITY −  SQL standard syntax for adding a row number to unnest() and

any other SRF call.

−  Authors: Andrew Gierth, David Fetter −  Reviewers: Dean Rasheed, Jeevan Chalke −  Committer: Greg Stark

SQL


-- Un-nest an array, and add a row number postgres=# SELECT * FROM unnest('{a,b,c,d,e,f}'::text[]) WITH ORDINALITY; unnest | ordinality --------+------------ a | 1 b | 2 c | 3 d | 4 e | 5 f | 6 (6 rows)

UNNEST/WITH ORDINALITY - Example


•  WITHIN GROUP −  This patch introduces generic support for ordered-set and

hypothetical-set aggregate functions, as well as implementations of the instances defined in SQL:2008 (percentile_cont(), percentile_disc(), rank(), dense_rank(), percent_rank(), cume_dist()). Also included are mode() (though it is not in the spec), as well as versions of percentile_cont() and percentile_disc() that can compute multiple percentile values in one pass over the data.

−  Authors: Atri Sharma, Andrew Gierth −  Reviewers: Pavel Stehule, Vik Fearing −  Committer: Tom Lane

SQL


-- Create a test table postgres=# CREATE TABLE t AS SELECT generate_series(1, 20) v; SELECT 20 -- Find the most common value postgres=# SELECT mode() WITHIN GROUP (ORDER BY v) FROM t; mode ------ 1 (1 row) -- Insert another (duplicate) value postgres=# INSERT INTO t (v) VALUES (15); INSERT 0 1 -- Now find the most common value postgres=# SELECT mode() WITHIN GROUP (ORDER BY v) FROM t; mode ------ 15 (1 row)

WITHIN GROUP – Example: mode()


-- Create a test table postgres=# CREATE TABLE t AS SELECT generate_series(1, 20) v; SELECT 20 -- Discrete Percentile - Select the first value whose position -- in the ordering equals or exceeds the specified fraction postgres=# SELECT percentile_disc(0.4) WITHIN GROUP (ORDER BY v) AS q FROM t; q --- 8 (1 row) -- Continuous Percentile - Select a value corresponding to the -- specified fraction in the ordering, interpolating if -- necessary postgres=# SELECT percentile_cont(0.4) WITHIN GROUP (ORDER BY v) AS q FROM t; q ----- 8.6 (1 row)

WITHIN GROUP – Example: percentile…()


Security


•  Remove Kerberos 5 authentication −  Does exactly what it says on the tin. Kerberos 5 authentication

has been deprecated since 8.3 - users should use GSSAPI instead.

−  Author: Magnus Hagander −  Committer: Magnus Hagander

Security


•  Various SSL/TLS improvements, including: −  Add TLS v1.1/1.2 support to the server −  Remove SSL v3 support. −  Change the default ciphersuite to HIGH:!aNULL

−  HIGH: Contains only secure and well-researched algorithms. −  !aNULL: Needed to disable suites that do not authenticate server.

−  Author: Marko Kreen −  Reviewers: Wim Lewis, Adrian Klaver −  Committers: Tom Lane, Peter Eisentratut, Noah Misch

Security


Miscellaneous


•  Dynamic shared memory −  This does not make shared_buffers dynamically resizeable!

What it does is allow the server to create new shared memory segments on the fly, to allow communication between processes. The driver for this is parallelism - e.g, for a backend to communicate with worker processes that it might spawn.

−  Authors: Robert Haas, Amit Kapila −  Committer: Robert Haas

Miscellaneous


•  Dynamic background workers −  This change allows new background workers to be started on

demand from a backend process, largely in preparation for future parallel query features. Background workers can be installed as extensions, and then started using an SQL function call (or from the backend code of course). The backend then signals the postmaster to start the required processes.

−  Author: Robert Haas −  Reviewer: Michael Paquier −  Committer: Robert Haas

Miscellaneous


And one more thing…


2014-03-15 22:35:15 EST LOG: loaded library "$libdir/plugins/plugin_debugger.so"




Ever seen this?


Lessen library-loading log level. author Jeff Davis <[email protected]> Sun, 24 Nov 2013 18:50:54 +0000 (10:50 -0800) committer Jeff Davis <[email protected]> Sun, 24 Nov 2013 18:50:54 +0000 (10:50 -0800) Previously, messages were emitted at the LOG level every time a backend preloaded a library. That was acceptable (though unnecessary) for shared_preload_libraries; but it was excessive for local_preload_libraries and session_preload_libraries. Reduce to DEBUG1. Also, there was logic in the EXEC_BACKEND case to avoid repeated messages for shared_preload_libraries by demoting them to DEBUG2. DEBUG1 seems more appropriate there, as well, so eliminate that special case. Peter Geoghegan.

Thank you Peter!


•  Depesz and Michael Paquier for the blogs

•  Thom Brown for comparing MongoDB to PostgreSQL

•  Developers for the awesome new features in 9.4

•  Reviewers for taking the time to test the patches

•  Committers for taking responsibility for the patches

Thank you…


Date post:	06-May-2015
Category:	Technology
Upload:	enterprisedb
View:	1,459 times
Download:	0 times

A Peek in the Elephant's Trunk

Technology