Postgresql 9.0 HA at LOADAYS 2012

;

PostgreSQL 9.0 HAJulien Pivotto

April, 1 2012 @ Loadays

;

OverviewPostgreSQL 9.0

ClusteringBackups

MonitoringAutomation

The end

The missionBefore the migration

Table of content1 Overview


2 PostgreSQL 9.0IntroStreaming replicationMaster configurationSlave configurationPostgreSQL specific tricksSetting up a slave

3 ClusteringSet up of corosyncOCF resource

4 BackupsCron jobsBackupPC

5 MonitoringNagiosMunin

6 AutomationPuppet moduleThe node file#TODO

7 The end

Julien Pivotto PostgreSQL 9.0 HA

;


ClusteringBackups


The end


Who am I

• Julien Pivotto• Consultant at Inuits since May 2011• FOSS defender since 2005


;


ClusteringBackups


The end


A.R.S.I.A.

• Association Régionale de Santé et d’Identification Animales• 30 linux servers in several locations• A lot of Open Source

• CentOS, Samba, Open-xchange, mailscanner, Cyrus,• . . . Puppet, jenkins, foreman, OpenVPN, GLPI, rabbitmq,• . . . BackupPC, CUPS, icinga, trac, zope, plone,• . . . solr, pentaho, funambol, munin, squid, asterisk,• . . . and PostgreSQL, . . .


;


ClusteringBackups


The end


C.E.R.I.S.E

• A web application• Plone (python)• 15k+ visits, 500k+ pages and 2.000.000+ hits each month• Developped by Affinitic

• Several databases• PostgreSQL 9.0• Oracle database

• Several servers/services• Two reverse proxies in failover HA• Two application servers in load balancing HA• Two PostgreSQL servers in failover HA• An oracledb server• A development server• A pentaho server

• Being integrated in jenkins (to be continued. . . )Julien Pivotto PostgreSQL 9.0 HA

;


ClusteringBackups


The end


PostgreSQL before the migration

• PostgreSQL 8.3.7• No native support of HA• High availability with heartbeat 2 and DRBD• Installed on the application servers• Nothing automated• Failover: Passive node is not even read only• Installed in November 2008


;


ClusteringBackups


The end


Monitoring before the installation

• Icinga• Check of the DRBD• Simple connection check to PostgreSQL

• Graphing with Cacti• Size of the databases• Connexions to the database• Checkpoints


;


ClusteringBackups


The end


Backups before the installation

• Backups were done every hour one the same machine• External backups once a day on disk and on tape• Backups are made with pg_dump command• BackupPC get those files


;


ClusteringBackups


The end

IntroStreaming replicationMaster configurationSlave configurationPostgreSQL specific tricksSetting up a slave

PostgreSQL 9.0

• PostgreSQL 9.0 was out in september 2010• It brings to the world native replication in PostgresSQL• There is not any native failover tool• So we need to use PostgreSQL + Corosync• The setup of PostgreSQL is managed by Puppet


;


ClusteringBackups


The end


Write-Ahead Logging

• It means that every change to datafile must first be writteninto a log file

• Less disk writes: only the log file needs to be flushed to disk toguarantee that a transaction is committed, rather than everydata file changed by the transaction


;


ClusteringBackups


The end


What is streaming replication

• Streaming replication provides the capability to ship and applyWAL XLOGS to standby servers

• It’s possible to have multiple standby servers• Standby servers can be read-only ("Hot standby")


;


ClusteringBackups


The end


DisadvantagesSpecifications of streaming replication

• Streaming replication supports only asynchronous log-shipping• But when the database is used, the delay is close to

synchronous log-shipping• Adding a standby server requires manual action

• But in our case we will only have one standby server• PostgreSQL does not provide HA feature

• But Corosync does• It is a single-threaded replication

• It is a single-threaded replication


;


ClusteringBackups


The end


Master configurationThe master only needs one configuration file.

Configuration non-related to SR#Postgresql configuration#http://www.postgresql.org/docs/9.0/interactive/index.htmllisten_addresses = ’*’max_connections = 200shared_buffers = 4096MBwork_mem = 4096MBeffective_cache_size = 10024MBcommit_delay = 100000effective_cache_size = 2560log_destination = ‘stderr’log_directory = ‘pg_log’logging_collector = onlog_filename = ‘postgresql-%Y-%m-%d_%H%M%S.log’log_truncate_on_rotation = onlog_rotation_age = 1dlog_rotation_size = 0log_min_messages = noticelog_min_duration_statement = 1000log_line_prefix = ‘%t %u ’log_statement = ‘none’datestyle = ‘iso, dmy’


;


ClusteringBackups


The end


Master configuration

Configuration related to SRwal_level = hot_standbymax_wal_senders = 2wal_keep_segments = 128


;


ClusteringBackups


The end


Master configuration

• wal_level = hot_standbyAllows stanby server to be readable

• max_wal_senders = 2We allow up to 2 standby nodes

• wal_keep_segments = 128The minimum wal segments to keep


;


ClusteringBackups


The end


Slave configuration

• The slave requires at least two configuration files• A postgreSQL.conf file• A recovery.conf file, used to apply the WAL XLOGS shipped by

the master

• A trigger file to stop replication can be specified

PostgreSQL.conf - Configuration related to SRwal_level = hot_standbyhot_standby = on

Note that the file also have the same first part of the config filethan the master configuration.


;


ClusteringBackups


The end


Slave configuration

recovery.confstandby_mode = ‘on’primary_conninfo = ‘host=192.168.177.2 user=replicuser’

• standby_mode means that this is a standby server• primary_conninfo is the connection to the master


;


ClusteringBackups


The end


Replication user

• A super user called replication has to be created• The SQL command to create it isCREATE USER replication SUPERUSER LOGIN CONNECTIONLIMIT 1 ENCRYPTED PASSWORD ‘foobar’;


;


ClusteringBackups


The end


pg_hba.conf

• pg_hba.conf is the file that contains some kind of ACLs forthe PostgreSQL connections

• In that file we will add both nodes as ‘trusted’ and thereplication user as trusted too

pg_hba.confhostnossl all all 10.0.10.8/32 trusthostnossl all all 10.0.10.9/32 trusthostnossl replication replicuser 192.168.177.2/24 trust


;


ClusteringBackups


The end


Setting up a slave

• You have to type a bunch of commands on the master whenyou add a new standby server

Adding a standby serverpsql -c "SELECT pg_start_backup(’label’, true)"rsync -a ${PGDATA}/ standby:/srv/pgsql/standby/ --exclude postmaster.pid --exclude ‘*-master’--exclude ‘*-slave’psql -c "SELECT pg_stop_backup()"


;


ClusteringBackups


The end

Set up of corosyncOCF resource

Corosync configuration

• The goal of corosync is to make the switch betweenmaster/slave when needed

• It will ensure that a master is online and connected to therouter

• The two servers are connected to each other on eth1• Corosync is installed by Puppet• We take it from the clusterlabs repositories• We use a personalized master/slave ocf resource to managethe PostgreSQL M/S


;


ClusteringBackups


The end


crm.conf

The main configuration file of corosync is/etc/corosync/crm.conf. It contains all theresources/nodes/etc. . .

Defining the nodesnode babar.interne.arsia.be \attributes standby="off"

node dumbo.interne.arsia.be \attributes standby="off"

In this code, the two nodes are defined, and we tell corosync thatthey should be started at launch.


;


ClusteringBackups


The end


crm.conf

Defining the primitivesprimitive pgsql ocf:inuits:pgsql-msprimitive virt_ip ocf:heartbeat:IPaddr2 \params nic="eth0" iflabel="0" ip="10.0.10.10" cidr_netmask="24" broadcast="10.0.10.255" \meta target-role="Started" is-managed="true"

primitive ping ocf:pacemaker:ping \params host_list="10.0.10.1" \op monitor interval="10s" timeout="10s" \op start interval="0" timeout="45s" \op stop interval="0" timeout="50s"

• We define 3 primitives:• pgsql, the PostgreSQL primitive• virt_ip, the floating IP address• ping, the primitive that will check that the servers are

connected to the router


;


ClusteringBackups


The end


crm.conf

Configuring the primitivesms pgsql-ms pgsql \params pgsqlconfig="/var/lib/pgsql/data/postgresql.conf" \lsb_script="/etc/init.d/postgresql-9.0" \pgsqlrecovery="/var/lib/pgsql/data/recovery.conf" \meta clone-max="2" clone-node-max="1" master-max="1" master-node-max="1" notify="false"

clone clone-ping ping \meta globally-unique="false"

• We configure the PostgreSQL M/S: the init script, theconfiguration files. . .

• We also configure the ping resource as a clone (it will belaunched on both servers)


;


ClusteringBackups


The end


crm.conf

Defining the nodesgroup PSQL virt_iplocation connected PSQL \rule $id="connected-rule" -inf: not_defined pingd or pingd lte 0

colocation ip_psql inf: PSQL pgsql-ms:Masterproperty $id="cib-bootstrap-options" \cluster-infrastructure="openais" \expected-quorum-votes="2" \stonith-enabled="false" \no-quorum-policy="ignore" \default-resource-stickiness="INFINITY"

rsc_defaults $id="rsc_defaults-options" \migration-threshold="INFINITY" \failure-timeout="10" \resource-stickiness="INFINITY"

• These lines will ensure that the master is always on the samenode as the floating IP address

• And also that the master is connected to the router


;


ClusteringBackups


The end


OCF resource

• There is a custom OCF resource to manage the master/slavePostgreSQL

• It is based on an example of resource written by AndrewBeekhof from Clusterlabs

• The file has to be in/usr/lib/ocf/resource.d/inuits/pgsql-ms


;


ClusteringBackups


The end


OCF resource

• The script does the following:• It moves the postgresql.conf-master topostgresql.conf when a node is promoted/master

• It moves the postgresql.conf-slave to postgresql.confwhen a node is depromoted/slave

• It ensure that recovery.conf-slave is on recovery.confon slave and absent on master

• It starts/restarts PostgreSQL when needed.

• I will post that file on Github soon


;


ClusteringBackups


The end

Cron jobsBackupPC

Backups of the databases

• Sometimes, you need backups (especially when you don’t havebackups. . . )

• We do a backup per hour on each node (one at minute 0 andone at minute 30)

• We do a backup per day on each node• We do a backup per day on before BackupPC backup on eachnode.

• We keep 24 hourly backups and 7 daily backups on disk• With BackupPC we keep months of backups


;


ClusteringBackups


The end

Cron jobsBackupPC

Hourly backup script

/usr/local/bin/backup_hourly.sh#!/bin/bashDATE=$(date +%H)BACKUP_PATH=/var/lib/backups/hourlyfor db in foobar_db foobar2_dbdo/usr/bin/pg_dump $db | gzip > $BACKUP_PATH/${db}_$DATE.pgsql.gzln -fs $BACKUP_PATH/${db}_$DATE.pgsql.gz $BACKUP_PATH/${db}_current.pgsql.gz

done

The daily script is almost the same.


;


ClusteringBackups


The end

Cron jobsBackupPC

BackupPC script

/usr/local/bin/backup_backuppc.sh#!/bin/bashDATE=$(date +%u)BACKUP_PATH=/var/lib/backups/backuppcfor db in cerise trackitquality trackit zodb_cerisedo/usr/bin/pg_dump -U postgres $db | gzip > $BACKUP_PATH/${db}_$DATE.pgsql.gzln -fs $BACKUP_PATH/${db}_$DATE.pgsql.gz $BACKUP_PATH/${db}_current.pgsql.gz

done

In the backupPC config, I added the following:

BackupPC config$Conf{DumpPreUserCmd} = ‘$sshPath -t -q -x -l backuppc $host /usr/local/bin/backup_backuppc.sh’;


;


ClusteringBackups


The end

NagiosMunin

check_postgres script

• The check_postgres.pl is a nagios-compatible perl script• Available on http://www.bucardo.org/check_postgres/and on Github

• What we check with it:• The current connections• The status of the replication (the delay)


http://www.bucardo.org/check_postgres/

;


ClusteringBackups


The end

NagiosMunin

Check hot_standby latency

• The check_postgres.pl script has a check for hot_standbydelay

• But we do not know who is the master and the slave, and it isrequired to launch the script

• So, here is a bash script I wrote to know the M/S order

Master/slave replication check#!/bin/bash/usr/lib64/nagios/plugins/check_postgres.pl --db="$1" \--action hot_standby_delay -w 300 -c 600 --host=$(crm_resource --resource pgsql-ms --locate|awk ‘/Master/ {master=$6} / $/ {slave=$6} END {print master","slave}’)


;


ClusteringBackups


The end

NagiosMunin

Munin postgres scripts

• Munin is shipped with perl plugins for postgresql• We use four of them:

• postgres_size,• postgres_checkpoints,• postgres_connections_db,• postgres_cache


;


ClusteringBackups


The end

NagiosMunin

Munin postgres scripts


;


ClusteringBackups


The end

Puppet moduleThe node file#TODO

Puppet module

• The puppet postgres module is forked from Kris Buytaert’sgithub page

• It is modified to remove all references to services, because wewant corosync to manage them

• It creates the users, the super users, the databases• It is a parameterized class, with a "cluster" parameter. So wecan also install simple PostgreSQL

• The cache sizes are parameterized too, so we can also use thatin Vagrant boxes

• Here are some examples from the module I will upload onGithub ASAP


;


ClusteringBackups


The end


Class postgres

The postgres class installs the packages and makes the initdb stuff.

init.ppclass postgres ($cluster = ‘no’,$running_ip = ‘127.0.0.1’

){ ...

• The cluster parameter indicates if we want or not clustering• running_ip is used for the SQL commands. In case of acluster, you have to put cluste’s IP address here.


;


ClusteringBackups


The end


Sqlexec definition

sqlexec.ppdefine postgres::sqlexec($username, $database, $sql, $sqlcheck) {exec{ "psql -h $postgres::running_ip –username=${username} $database

-c $̈{sql}>̈> /var/log/puppet-postgresql.sql.log 2>&1 && /bin/sleep 5":environment => "PGPASSWORD=${postgres_password}",path => $::path,timeout => 600,unless => "psql -h $postgres::running_ip -U $username $database -c $sqlcheck",require => Service[’postgresql-9.0’],

}}


;


ClusteringBackups


The end


Example in the node file

Here is the result in the node file:

dumbo.ppnode babar {

class {’postgres’:cluster => ’yes’,running_ip => ’10.0.10.10’,

}include postgres::munininclude postgres::backupinclude cluster::node

postgres::config{$::fqdn: listen => ’*’,

}


;


ClusteringBackups


The end



dumbo.pppostgres::hba {

$::fqdn:allowedrules => [

"host all all $::ipaddress/32 trust",’hostnossl all all 10.0.10.8/32 trust’,’hostnossl all all 10.0.10.9/32 trust’,’hostnossl all all 10.0.10.10/32 trust’,’hostnossl replication replicuser 192.168.177.2/24 trust’,

],}


;


ClusteringBackups


The end



dumbo.pppostgres::createsuperuser{

’replicuser’:passwd => ’foobar’,

}

postgres::createuser{’cerise’:passwd => ’foobar’;

}

postgres::createdb{’zodb_cerise’:owner => ’cerise’,require => Postgres::Createuser[’cerise’],

}}


;


ClusteringBackups


The end


#TODO

• The first synchronisation is not puppetized• More advanced checks on the database #monitoringsucks(e.g. slow queries)

• A disaster recovery• Improve the ocf script• Check the content of the backups• . . .


;


ClusteringBackups


The end

Any questions?


Date post:	06-May-2015
Category:	Technology
Upload:	julien-pivotto
View:	5,105 times
Download:	3 times

Postgresql 9.0 HA at LOADAYS 2012

Technology