Agenda Isolate Services Durability Availability Conclusion
Implementing High Availabilitywith PostgreSQL
Dimitri [email protected]
25 Oct. 2012
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Dimitri Fontaine
2ndQuadrant FrancePostgreSQL Major Contributor
pgloader, prefix, skytools, debian, . . .CREATE EXTENSION
CREATE EVENT TRIGGER
Bi-Directional ReplicationPartitioning
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Dimitri Fontaine
2ndQuadrant FrancePostgreSQL Major Contributor
pgloader, prefix, skytools, debian, . . .CREATE EXTENSION
CREATE EVENT TRIGGER
Bi-Directional ReplicationPartitioning
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Dimitri Fontaine
2ndQuadrant FrancePostgreSQL Major Contributor
pgloader, prefix, skytools, debian, . . .CREATE EXTENSION
CREATE EVENT TRIGGER
Bi-Directional ReplicationPartitioning
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Proven Architectures, implemented lots at Hi-Media
Hi-Media (turnover scale: 200 millions e)
3 different activities to get money from the web
Allopass, HiPay: internet (micro) paymentTelecom ServiceAdvertising
PostgreSQL is the heart of the technical platform
Business needs complianceCapacity to adapt to changes
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Proven Architectures, implemented lots at Hi-Media
Hi-Media (turnover scale: 200 millions e)
3 different activities to get money from the web
Allopass, HiPay: internet (micro) paymentTelecom ServiceAdvertising
PostgreSQL is the heart of the technical platform
Business needs complianceCapacity to adapt to changes
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Proven Architectures, implemented lots at Hi-Media
Hi-Media (turnover scale: 200 millions e)
3 different activities to get money from the web
Allopass, HiPay: internet (micro) paymentTelecom ServiceAdvertising
PostgreSQL is the heart of the technical platform
Business needs complianceCapacity to adapt to changes
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Proven Architectures, implemented lots at Hi-Media
Hi-Media (turnover scale: 200 millions e)
3 different activities to get money from the web
Allopass, HiPay: internet (micro) paymentTelecom ServiceAdvertising
PostgreSQL is the heart of the technical platform
Business needs complianceCapacity to adapt to changes
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
PostgreSQL: Your data is our job
How to ensure both duralibity and availability of your data?
Usual needs:
ReliabilityStabilityPerformancesGrowth capacity (think commercial success)Continuity and Innovation
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
PostgreSQL: Your data is our job
How to ensure both duralibity and availability of your data?
Usual needs:
ReliabilityStabilityPerformancesGrowth capacity (think commercial success)Continuity and Innovation
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
PostgreSQL: Your data is our job
How to ensure both duralibity and availability of your data?
Usual needs:
ReliabilityStabilityPerformancesGrowth capacity (think commercial success)Continuity and Innovation
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Glossary
Some vocabulary
AvailabilityDuralibity (ACID)ArchitecturesReplications
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Glossary
Some vocabulary
AvailabilityDuralibity (ACID)ArchitecturesReplications
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Glossary
Some vocabulary
AvailabilityDuralibity (ACID)ArchitecturesReplications
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Glossary
Some vocabulary
Availability of services or of data?Duralibity (ACID)ArchitecturesReplications
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Glossary
Some vocabulary
Availability of services or of data?Duralibity (ACID)ArchitecturesReplications
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Glossary
Some vocabulary
Availability of services or of data?Duralibity (ACID)ArchitecturesReplications
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Needs first
Needs evolve, solutions must adapt
Start simpleSome first classic stepsHigh Availability of DataHigh Availability of ServicesRead Only Load BalancingRead Write Load Balancing
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Needs first
Needs evolve, solutions must adapt
Start simpleSome first classic stepsHigh Availability of DataHigh Availability of ServicesRead Only Load BalancingRead Write Load Balancing
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Needs first
Needs evolve, solutions must adapt
Start simpleSome first classic stepsHigh Availability of DataHigh Availability of ServicesRead Only Load BalancingRead Write Load Balancing
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Let’s start simple
Our projet life cycle
Let’s start with the example of a quite simpleproject released as a web application seeing itsneeds evolve with its success.
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Scaling out 101
Services Availability
Front servers are statelessWatch out for max_connectionsDon’t you use persistent connections!pgbouncer
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Scaling out 101
Services Availability
Front servers are statelessWatch out for max_connectionsDon’t you use persistent connections!pgbouncer
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Scaling out 101
Services Availability
Front servers are statelessWatch out for max_connectionsDon’t you use persistent connections!pgbouncer
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Scaling out 101
Using more than a single server and a connection pool
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
pgbouncer
pgbouncer is able to reuse client and server side connections.
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
pgbouncer
pgbouncer is able to reuse client and server side connections.
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Getting serious: backups
Backup Strategy is the single most important step towards dataavailability
Nightly pg_dump -Fc
Don’t forget pg_dumpall–globals-only
Data Retention7 days of nightly backups7 weeks of weekly backups12 months of monthly backups30 years of yearly backups?
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Getting serious: backups
Backup Strategy is the single most important step towards dataavailability
Nightly pg_dump -Fc
Don’t forget pg_dumpall–globals-only
Data Retention7 days of nightly backups7 weeks of weekly backups12 months of monthly backups30 years of yearly backups?
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Getting serious: backups
Backup Strategy is the single most important step towards dataavailability
Nightly pg_dump -Fc
Don’t forget pg_dumpall–globals-only
Data Retention7 days of nightly backups7 weeks of weekly backups12 months of monthly backups30 years of yearly backups?
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Failover, 101
pg_dump, pg_restore
protection against errors and omissionsbeware of restoring timestill a must have for data durabilitywhat about data availability?
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Failover, 101
pg_dump, pg_restore
protection against errors and omissionsbeware of restoring timestill a must have for data durabilitywhat about data availability?
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Failover, 201
Using physical backups and Point In Time Recovery
Point In Time Recovery, 8.1warm standby, 8.2Archiving and crash recoveryarchive_command
restore_command
walmgr.py, WAL-E
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Failover, 201
Using physical backups and Point In Time Recovery
Point In Time Recovery, 8.1warm standby, 8.2Archiving and crash recoveryarchive_command
restore_command
walmgr.py, WAL-E
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Failover, 201
Using physical backups and Point In Time Recovery
Point In Time Recovery, 8.1warm standby, 8.2Archiving and crash recoveryarchive_command
restore_command
walmgr.py, WAL-E
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Failover, 201
Using physical backups and Point In Time Recovery
Point In Time Recovery, 8.1warm standby, 8.2Archiving and crash recoveryarchive_command
restore_command
walmgr.py, WAL-E
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Warm Standby
Implementing Warm Standby
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Application Split
When you have separate backoffice and production requirements
Cross replicationSlony, Londiste, BucardoSpecific processing, batchesOff-line processingStill Transactional processingSkytools comes with PGQ
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Application Split
When you have separate backoffice and production requirements
Cross replicationSlony, Londiste, BucardoSpecific processing, batchesOff-line processingStill Transactional processingSkytools comes with PGQ
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Application Split
When you have separate backoffice and production requirements
Cross replicationSlony, Londiste, BucardoSpecific processing, batchesOff-line processingStill Transactional processingSkytools comes with PGQ
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Application Split
When you have separate backoffice and production requirements
Cross replicationSlony, Londiste, BucardoSpecific processing, batchesOff-line processingStill Transactional processingSkytools comes with PGQ
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Application Split
Implementing londiste and PGQ
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Queueing with PGQ
Off-line processing is better done with PGQ
Mainly written in PLpgSQL (and C)Client API for pythonand PHP
some work is happening for JavaCooperative Worker (Skytools 3)
PGQ: Stable, Reliable, Easy to monitor
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Queueing with PGQ
Off-line processing is better done with PGQ
Mainly written in PLpgSQL (and C)Client API for pythonand PHP
some work is happening for JavaCooperative Worker (Skytools 3)
PGQ: Stable, Reliable, Easy to monitor
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Queueing with PGQ
Off-line processing is better done with PGQ
Mainly written in PLpgSQL (and C)Client API for pythonand PHP
some work is happening for JavaCooperative Worker (Skytools 3)
PGQ: Stable, Reliable, Easy to monitor
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Queueing with PGQ
Off-line processing is better done with PGQ
Mainly written in PLpgSQL (and C)Client API for pythonand PHP
some work is happening for JavaCooperative Worker (Skytools 3)
PGQ: Stable, Reliable, Easy to monitor
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Disaster Recovery and Service Continuity
PostgreSQL 9.1 offers Synchronous Replication and Hot Standby
Hot Standby(A)Synchronous ReplicationStandby connects with libpq
recovery.conf
Continue ArchivingSwitchabe per transactionGood performance level
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Disaster Recovery and Service Continuity
PostgreSQL 9.1 offers Synchronous Replication and Hot Standby
Hot Standby(A)Synchronous ReplicationStandby connects with libpq
recovery.conf
Continue ArchivingSwitchabe per transactionGood performance level
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Disaster Recovery and Service Continuity
PostgreSQL 9.1 offers Synchronous Replication and Hot Standby
Hot Standby(A)Synchronous ReplicationStandby connects with libpq
recovery.conf
Continue ArchivingSwitchabe per transactionGood performance level
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Disaster Recovery and Service Continuity
PostgreSQL 9.1 offers Synchronous Replication and Hot Standby
Hot Standby(A)Synchronous ReplicationStandby connects with libpq
recovery.conf
Continue ArchivingSwitchabe per transactionGood performance level
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Disaster Recovery and Service Continuity
Implementing Hot Standby
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Scaling Writes
PL/proxy
Scale-up or Scale-out?Remote Procedure CallShardingDistributed databaseAutonomous TransactionsStored Procedures
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Scaling Writes
PL/proxy
Scale-up or Scale-out?Remote Procedure CallShardingDistributed databaseAutonomous TransactionsStored Procedures
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Scaling Writes
PL/proxy
Scale-up or Scale-out?Remote Procedure CallShardingDistributed databaseAutonomous TransactionsStored Procedures
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Scaling Writes
PL/proxy
Scale-up or Scale-out?Remote Procedure CallShardingDistributed databaseAutonomous TransactionsStored Procedures
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Scaling Writes
PL/proxy
Scale-up or Scale-out?Remote Procedure CallShardingDistributed databaseAutonomous TransactionsStored Procedures
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
PL/Proxy
Installing plproxy
Example (install.sql)=# create extension plproxy;=# create extension hashlib;
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
PL/Proxy
pl/proxy is the integrated sharding layer. Now you have to write allyour SQL in server side functions.
Example (admin/change_group_status.sql)create or replace function admin.change_group_status(
user_name text, status integer)returns void as $BODY$
CLUSTER ’fl_cluster’;RUN ON hash_string(user_name, ’lookup3le’);
$BODY$;
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Distributed Sequences 1/2
Same schema on every node, best to avoid overlapping sequences.
Example (distributing sequence)ALTER SEQUENCE foo_id_seq
INCREMENT BY 16;
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Distributed Sequences 2/2
Same schema on every node, best to avoid overlapping sequences.
Example (distributing sequence)ALTER SEQUENCE foo_id_seq
MINVALUE 297000000000000000MAXVALUE 297999999999999999;
http://www.fotolog.com/<user>/297000000000017139/
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Distributed Sequences 2/2
Same schema on every node, best to avoid overlapping sequences.
Example (distributing sequence)ALTER SEQUENCE foo_id_seq
MINVALUE 297000000000000000MAXVALUE 297999999999999999;
http://www.fotolog.com/<user>/297000000000017139/
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Scaling Writes
Implementing plproxy
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Scaling Writes
Implementing plproxy
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Distributed High Availability
Retrospective and Future of PostgreSQL Replication
8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Distributed High Availability
Retrospective and Future of PostgreSQL Replication
8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Distributed High Availability
Retrospective and Future of PostgreSQL Replication
8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Distributed High Availability
Retrospective and Future of PostgreSQL Replication
8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Distributed High Availability
Retrospective and Future of PostgreSQL Replication
8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Distributed High Availability
Retrospective and Future of PostgreSQL Replication
8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Distributed High Availability
Retrospective and Future of PostgreSQL Replication
8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication
Dimitri [email protected] Implementing High Availability
Agenda Isolate Services Durability Availability Conclusion
Questions?
Meet with us on the booth, join us in the Hallway Track!
Want to win a blue elephant?http://2012.pgconf.eu/feedback/
Dimitri [email protected] Implementing High Availability