LET’S UNDERSTAND THE EPHEMERAL NATURE OF CONTAINER / POD
ephemeral | ɪˈfɛm(ə)r(ə)l, ɪˈfiːm(ə)r(ə)l |
adjectivelasting for a very short time
STATELESS BY NATUREAny data saved within the
pod, is available till its lifetime
IP ADDRESSEverytime a pod is
scheduled, it comes with a new IP
PID 1Container lives as long as
PID 1 lives
PEERSA pod can be disturbed
by its neighbours
THINGS NEEDS TO RUN AN INSTANCE OF POSTGRESQL
NON ROOT USERA non root user is need
to run PostgreSQL
HOST BASED CONFIGURATIONNeed to configure
pg_hba.conf file to allow other hosts to communicate
POSTGRES CONFIGURATIONConfigurations to tweak PostgreSQL for custom
needs
OPERATING SYSTEMTo run PostgreSQL
database
POSTGRES BINARYBased on the operating system, corresponding
package manager is used to install PostgreSQL
EXTERNAL DISKExternal disk to store the
PostgreSQL data
ADDITIONAL THINGS NEEDS TO RUN HIGHLY AVAILABLE POSTGRESQL
REPLICATION USERA non root user is need
to run PostgreSQL
HOST BASED CONFIGURATIONNeed to configure
pg_hba.conf file to allow other secondary hosts to
communicate
POSTGRES CONFIGURATIONMake necessary changes
to PostgreSQL configuration for
replication
PERFORM BASE BACKUPCopy the data dir to all the secondary instances to begin the replication
then on
■ Kubernetes offers all the necessary building block but it’s still difficult to run and maintain PostgreSQL on it.
■ Given that Kubernetes is meant to run ephemeral loads, downtime could happen at any time if we run it with just the basic blocks offered by Kubernetes.
WHAT IS MISSING?
“To be clear Kubernetes does support stateful workloads.
Kubernetes has StatefulSets, Volumes, and many other
features to ease the burden of running stateful workloads. But
you still require expertise to effectively manage stateful
services on Kubernetes.”
— Kelsey Hightower, Google
We need a process manager to- generate PostgreSQL configuration for all the instance based on its role (primary vs
secondary)- restart / reload PostgreSQL based on the data available on the store
POSTGRES PROCESS MANAGER2
We need a proxy to hide which one is primary, which one is secondary so that client doesn’t have to keep switching
PROXY4
LET’S INTRODUCE SOME TERMINOLOGY
PROXY -> PROXY4
CENTRALIZED STORE -> CONFIG STORE1
POSTGRES PROCESS MANAGER -> KEEPER2
MONITOR MASTER HEALTH -> SENTINEL3
REQUIREMENTS FOR CONFIG STORE
■ It should highly available by default without any additional effort
■ It should be distributed by default
■ It should be highly performant
CONFIG STORE
Raft based distributed KV store from CoreOS (now part of
Redhat, which is now part of IBM)
ETCDRaft based distributed KV
Store from Hashicorp
CONSULWe can leverage
Kubernetes Config Map as distributed store
K8s CONFIG MAP
FINAL ARCHITECTURE
Sentinel(elected)
Keeper(secondary)
Proxy
Client ClientClient
Keeper(secondary)
Keeper(primary)
ProxyProxy
SentinelSentinel
OPEN SOURCE SOLUTION
Is a template for you to create your own customized, high-availability solution using Python and - for maximum
accessibility
PATRONIIs a cloud native
PostgreSQL manager for PostgreSQL high
availability
STOLON
Steps:1. Setup streaming replication with standby cluster created with SSD2. Once the replication fully sync get rid of the old stateful set and underlying PVC
MIGRATING FROM HDD -> SSD2
Steps:1. Scale down the stateful set2. Backup disk3. Create a job to mount the PVC used by the statefulset4. Run the pg_upgradecluster to migrate
UPGRADING POSTGRESQL VERSION3
K8s config store availability is greatly dependent on how K8s cluster is setup
USE K8s CONFIG STORE WITH CAUTION1
With VMs we generally run single instance of PostgreSQL, in case of Kubernetes there could be other workloads scheduled by the orchestrator and it could interfere with PostgreSQL performance
NEIGHBOUR NOISE2
Kubernetes scheduler needs this information to schedule your pod accordingly
ALWAYS SPECIFY POD RESOURCE REQUEST3
If you run them together there will too much neighbour noise
HAVE DIFFERENT K8s CLUSTER FOR STATELESS AND STATEFUL APPS4
Even though it is possible to setup PostgreSQL HA, ensure to backup the disk in regular interval
TAKE DAILY BACKUP OF DISK5
Thanks to
◂ Presentation template by Slidesgo◂ Icons by Flaticon◂ Infographics by Freepik◂ Images created by Freepik - Freepik
CREDITS