Amazon RDS PostgreSQL: Enabling Innovation with Cloud Managed Databases

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Grant McAlister – Senior Principal Engineer - RDS

March 2017

Amazon RDS for PostgreSQLEnabling Innovation with Cloud Managed Databases

Development to Production

DB1

Test1

DB2

Test2

Big Test

DB1 DB2 DB3

DB4 DB5 DB6

Production

DB1DB2

QA

DB1 DB2

Patching

HA & DRBackup & Recovery

ScaleSecurity

Amazon Relational Database Service

RDS is a managed Relational database service that is simple to deploy, easy to scale, reliable and cost-effective

Managed Service

Easy to Scale and Operate

Choice of Database Engines

High Availability

High Performance

Amazon Relational Database Service (RDS)

RDS Version Updates

Now Supporting 9.6

Minor Releases• 9.6.1• 9.5.4 • 9.4.9 • 9.3.14

Extension Support Additions

9.6 bloom & pg_visibility

[email protected]

9.3 Original - 32

9.3 Current - 35

9.4 Current - 399.5 Current - 44

Future - ???9.6 Current - 46

Availability

Backups and Disaster RecoveryDB Snapshots

• User-driven snapshots of database• Kept until explicitly deleted

Automated Backups• Nightly system snapshots + transaction backup• Enables point-in-time restore to any point in retention period, up to

the last 5 minutes• Max retention period = 35 days

Cross region copy• Change regions• Disaster recovery

Availability – Read and Write – Multi-AZ

Secondary

Physical Synchronous

Replication

Primary

AZ1 AZ2

Application Application


SecondarySecondary Primary


Replication

AZ1 AZ2



SecondarySecondary Primary


Replication

AZ1 AZ2

DNS

cname update


Primary Update


Secondary Primary


Replication

AZ1 AZ2

DNS

cname update


Read Replicas = Availability

Secondary

Application

Read Replica

Read Replica

SyncReplication

Multi-AZ

Primary

Async Replication


Secondary

Application

Read Replica

Read Replica

SyncReplication

Multi-AZ

Primary

Eventually Consistent Reads

Writes & Consistent

Reads

Async Replication


Application

Read Replica

Read ReplicaSecondary Primary


Writes & Consistent

Reads

Async Replication


Application

Read Replica



Writes & Consistent

Reads

Async Replication


Application

Read Replica



Writes & Consistent

Reads

Async Replication

Upgrade


Application

Read Replica



Writes & Consistent

Reads

Async Replication

Modify

Cross Region Replicas – DR & Moves

SecondarySync

Application

Read ReplicaPrimary

AZ1 AZ2

Application

Read Replica

AZ1Async Replication

US-EAST-1 EU-WEST-1


SecondarySync

Application

Read ReplicaPrimary

AZ1 AZ2

Application

Read Replica


US-EAST-1 EU-WEST-1


SecondarySync

Application

Read ReplicaPrimary

AZ1 AZ2

Application

Read Replica

AZ1

New Primary

Async Replication

US-EAST-1 EU-WEST-1


SecondarySync

Application

Read ReplicaPrimary

AZ1 AZ2

Application

Read Replica

Application

AZ1

New Primary

Async Replication

US-EAST-1 EU-WEST-1


SecondarySync

Application

Read ReplicaPrimary

AZ1 AZ2

Application

Read Replica

Application

AZ1

New Primary

Async Replication

US-EAST-1 EU-WEST-1

Application

SecondarySync

AZ2


Read Replica

Application

AZ1

New Primary

US-EAST-1 EU-WEST-1

Application

SecondarySync

AZ2

Cross Region Replicas – Reduce Latency

SecondarySync

Application

Read ReplicaPrimary

AZ1 AZ2

Application

US-EAST-1

Cross Region Replicas – Reduce Latency

SecondarySync

Application

Read ReplicaPrimary

AZ1 AZ2

Application

Read Replica

Application


US-EAST-1 EU-WEST-1

AuroraR/W


Aurora R

Application ROApplication

SyncReplication

AZ-1 AZ-2 AZ-3

Storage Storage Storage Storage Storage Storage

Aurora Storage

4/6 sync writes

Aurora PostgreSQL – In Preview

AuroraR/W


Aurora R


SyncReplication

AZ-1 AZ-2 AZ-3


Aurora Storage

4/6 sync writes


Async Invalidation

AuroraR/W


Aurora R


SyncReplication

AZ-1 AZ-2 AZ-3


Aurora Storage

4/6 sync writes

AuroraR/W

Aurora R


Async Invalidation

FAILOVER

Amazon Aurora with PostgreSQL Compatibility

Performance By The NumbersMeasurement Result

PgBench >= 2x faster

SysBench 2x-3x faster

Data Loading 3x faster

Response Time >2x faster

Throughput Jitter >3x more consistent

Throughput at Scale 3x faster

Recovery Speed Up to 85x faster

Patching

Minor Version Patching

• Customer Control Patching• Execute immediately or in Weekly Maintenance Window

• Auto Patching can be enabled • Allows for hands off upgrade in next Maintenance Window

Major version upgrade

Prod9.5


Prod9.5

Prod9.6

pg_upgrade

Backup Backup

No PITR


Prod9.5

Test9.5

Restore to a test instance


Prod9.5

Test9.5

pg_upgrade



Prod9.5

Test9.5

Test9.6

pg_upgrade



Prod9.5

Test9.5

Test9.6

pg_upgrade


Application Testing


Prod9.5

Test9.5

Test9.6

pg_upgrade


Application Testing


Prod9.5

Prod9.6

pg_upgrade

Backup Backup

No PITR

Test9.5

Test9.6

pg_upgrade


Application Testing

Security

Forcing SSL on all connections

DB Instance

SnapshotApplication

Host

Log Backups


DB Instance

SnapshotApplication

Host

Log Backups

Security Group


DB Instance

SnapshotApplication

HostSSL

Log Backups

Security Group


DB Instance

SnapshotApplication

HostSSL

Log Backups

Security Group

VPC


DB Instance

SnapshotApplication

HostSSL

Log Backups

Security Group

VPC

Encryption at Rest


DB Instance

SnapshotApplication

HostSSL

Log Backups

Security Group

VPC

Encryption at Rest

ssl_mode=disable


DB Instance

SnapshotApplication

HostSSL

Log Backups

Security Group

VPC

Encryption at Rest

ssl_mode=disable


DB Instance

SnapshotApplication

HostSSL

Log Backups

Security Group

VPC

Encryption at Rest

ssl_mode=disable

rds.force_ssl=1 (default 0)


DB Instance

SnapshotApplication

HostSSL

Log Backups

Security Group

VPC

Encryption at Rest

ssl_mode=disable

rds.force_ssl=1 (default 0)

Unencrypted Snapshot Sharing

DB Instance

Snapshot

Prod Account

Test Account

Snapshot

Share with account


DB Instance

Snapshot

Prod Account

Test Account

Snapshot

Snapshot

Share with account


DB Instance

Snapshot

Prod Account

Test Account

SnapshotDB Instance

Snapshot

Share with account


DB Instance

Snapshot

Prod Account

Test Account

SnapshotDB Instance

Snapshot

Share with account

Share to Public

Encrypted Snapshot Sharing

DB Instance

Snapshot

Prod Account

Test Account

Encryption at Rest

Default


DB Instance

Snapshot

Prod Account

Test Account

Snapshot

Share with account

Encryption at Rest

Default


DB Instance

Snapshot

Prod Account

Test Account

Snapshot

Share with account

Encryption at Rest

Default


DB Instance

Snapshot

Prod Account

Test Account

Snapshot

Share with account

Encryption at Rest


DB Instance

Snapshot

Prod Account

Test Account

Snapshot

Share with account

Encryption at Rest

CustomKey


DB Instance

Snapshot

Prod Account

Test Account

Snapshot

Share with account

Encryption at Rest

CustomKey

Add external account


DB Instance

Snapshot

Prod Account

Test Account

SnapshotDB Instance

Snapshot

Share with account

Encryption at Rest

CustomKey

Add external account

Cross Region Replicas – Encrypted

SecondarySync

Application

Primary

AZ1 AZ2

Application

US-EAST-1

Cross Region Replicas – Encrypted

SecondarySync

Application

Primary

AZ1 AZ2

Application

Read Replica

Application


US-EAST-1 EU-WEST-1

2 Threads 4 Threads 8 Threads 16 Threads 32 Threads 64 Threads0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500PG Bench - Read & Write

RegularEncrypted

Tran

sact

ions

Per

Sec

ond

(TPS

)Encryption at rest overhead

5 to 10% Overhead on heavy write

HIPAA-eligible service & FedRAMP

• RDS PostgreSQL is now a HIPAA-eligible service• https://aws.amazon.com/compliance/hipaa-compliance/

• FedRAMP in AWS GovCloud (US) region • https://aws.amazon.com/compliance/fedramp/

Data movement

Move data to the same or different database engine Keep your apps running during the migrationStart your first migration in 10 minutes or lessReplicate within, to, or from AWS EC2 or RDS

AWSDatabase Migration

Service(DMS)

CustomerPremises

Application Users

EC2or

RDS

Internet

VPN

Start a replication instance

Keep your apps running during the migration

AWS Database Migration Service

CustomerPremises

Application Users

EC2or

RDS

Internet

VPN


Connect to source and target databases

Select tables, schemas, or databases



CustomerPremises

Application Users

EC2or

RDS

Internet

VPN




Let the AWS Database Migration Service create tables and load data



CustomerPremises

Application Users

EC2or

RDS

Internet

VPN




Let the AWS Database Migration Service create tables and load data Uses change data capture to keep them in sync



CustomerPremises

Application Users

EC2or

RDS

Internet

VPN




Let the AWS Database Migration Service create tables and load data Uses change data capture to keep them in syncSwitch applications over to the target at your convenience



AWS Database Migration Service - PostgreSQL

• Source - on premise or EC2 PostgreSQL (9.4+) RDS (9.4.9+ or 9.5.4+ or 9.6.1)

• Destination can be EC2 or RDS• Initial bulk copy via consistent select• Uses PostgreSQL logical replication support to provide

change data capture

https://aws.amazon.com/dms/

https://aws.amazon.com/dms/

Logical Replication Support• Supported with 9.6.1+, 9.5.4+ and 9.4.9+• Set rds.logical_replication parameter to 1• As user who has rds_replication & rds_superuser role

SELECT * FROM pg_create_logical_replication_slot('test_slot', 'test_decoding');

pg_recvlogical -d postgres --slot test_slot -U master --host $rds_hostname -f - --start

• Added support for Event Triggers

Logical Decoding Space Usage

CloudWatch – Replication Lag

CloudWatch – Slot usage for WAL

Logical Replication Support - Example

RDS Postgres

RDS Postgres

Logical Replica

Redshift

DMS


RDS Postgres

RDS Postgres

Logical Replica

Redshift

On PremisePostgres

DMS


RDS Postgres

RDS Postgres

Logical Replica

RedshiftEC2 Postgres

On PremisePostgres

DMS


RDS Postgres

RDS Postgres

Logical Replica


On PremisePostgres

DMS

EC2 Oracle


RDS Postgres

RDS Postgres

Logical Replica


On PremisePostgres

DMS

EC2 Oracle

CustomLogicalHandler

NoSQL DB

Schema Conversion Tool - SCT

Downloadable tool (Windows, Mac, Linux Desktop)

Source Database Target Database on Amazon RDSMicrosoft SQL Server Amazon Aurora, MySQL, PostgreSQLMySQL PostgreSQLOracle Amazon Aurora, MySQL, PostgreSQLPostgreSQL Amazon Aurora, MySQL

SCT - Analysis

SCT - Detailed

Scale & Performance

Push Button Scaling

Scale nodes vertically up or down• db.t2.micro (1 virtual cores, 1GB)• db.m4.large (2 virtual cores, 8GB) • db.r3.8xlarge(32 virtual cores, 244GB)

• db.r4.16xlarge (64 virtual cores, 488GB) Aurora

Storage Scaling EBS Based - 5GB-6TB & Up to 30,000 IOPSAurora Based – Up to 64 TB

M4 Instance Class – pgbench read only

1 2 4 8 160

2000

4000

6000

8000

10000

12000

14000

db.m3.large db.m4.large

Threads

Tran

sact

ions

per

Sec

ond

(TPS

)

46% Better Price/Performance

37% TPS Increase

$0.195 $0.182

Enhanced Operating System (OS) metrics

1-60 second granularity

cpuUtilization• guest• irq• system• wait• idl: • user • total • steal • nice

diskIO • writeKbPS• readIOsPS• await • readKbPS• rrqmPS • util • avgQueueLen • tps • readKb • writeKb • avgReqSz • wrqmPS • writeIOsPS

memory • writeback• cached • free • inactive• dirty • mapped • active • total • slab • buffers• pageTable• Hugepages

swap • cached • total • free

tasks• sleeping • zombie • running • stopped • total • blocked

fileSys • used• usedFiles• usedFilePercent• maxFiles • total • usedPercent

loadAverageMinute • fifteen • five • one

uptime

processList• name• cpuTime• parentID• memoryUsedPct• cpuUsedPct• id • rss• vss

Process List

OS metrics

Performance Insights – In Preview

Performance Insights – In Preview

Stats on RAMDISK

• Set rds.pg_stat_ramdisk_size in MB’s• Creates a RAM disk and sets stats_temp_directory to

use it.

• Reduces IOPS

• Good for instances with many tables/indexes and databases.

Burst mode: GP2 and T2

T2 – Amazon EC2 instance with burst capability• Base performance + burst • Earn credits per hour when below base performance• Can store up to 24 hours worth of credits• Amazon CloudWatch metrics to see credits and usage

GP2 – SSD-based Amazon EBS storage• 3 IOPS per GB base performance• Earn credits when usage below base• Burst to 3000+ IOPS

T2 – CPU credits

CPU credit balance

T2 exhaust CPU credits

Burst mode: what’s new

db.t2.large• 60 CPU Initial Credit• 36 CPU Credit earned per hour• Base Performance – 60%• 8 GB RAM• Increased IO bandwidth• Encryption at rest support

Future – t2.xlarge & t2.2xlarge

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 240

2000

4000

6000

8000

10000

12000100% Read - 20GB data

db.m1.medium + 200GB standarddb.m3.medium + 200G + 2000 IOPSdb.m3.large + 200G + 2000 IOPSdb.t2.medium + 200GB gp2

Hours

Tran

sact

ions

per

Sec

ond

(TPS

)Burst mode vs. Classic vs. Provisioned IOPS

$0.10 per hour

$0.58 per hour$0.40 per hour

$0.50 per hour

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 240

2000

4000

6000

8000

10000

12000100% Read - 20GB data

db.m1.medium + 200GB standarddb.m3.medium + 200G + 2000 IOPSdb.m3.large + 200G + 2000 IOPSdb.t2.medium + 200GB gp2db.t2.medium + 1TB gp2

Hours

Tran

sact

ions

per

Sec

ond

(TPS


$0.10 per hour

$0.58 per hour

$0.23 per hour

$0.40 per hour

$0.50 per hour

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 240

2000

4000

6000

8000

10000

12000100% Read - 20GB data

db.m1.medium + 200GB standarddb.m3.medium + 200G + 2000 IOPSdb.m3.large + 200G + 2000 IOPSdb.t2.medium + 200GB gp2db.t2.medium + 1TB gp2db.t2.large + 1TB gp2

Hours

Tran

sact

ions

per

Sec

ond

(TPS


$0.10 per hour

$0.58 per hour

$0.23 per hour

$0.40 per hour

$0.50 per hour

$0.30 per hour

Lessons

Vacuum parameters

Will auto vacuum when• autovacuum_vacuum_threshold +

autovacuum_vacuum_scale_factor * pgclass.reltuples

How hard auto vacuum works • autovacuum_max_workers• autovacuum_nap_time• autovacuum_cost_limit• autovacuum_cost_delay

Transaction IDWrap Around

RDS autovacuum logging (9.4.5+)

log_autovacuum_min_duration = 5000 (i.e. 5 secs)rds.force_autovacuum_logging_level = LOG

…[14638]:ERROR: canceling autovacuum task…[14638]:CONTEXT: automatic vacuum of table "postgres.public.pgbench_tellers"…[14638]:LOG: skipping vacuum of "pgbench_branches" --- lock not available

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Autovacuum



RDS autovacuum visibility(9.3.12, 9.4.7, 9.5.2, 9.6.1)pg_stat_activity

BEFORE usename | query----------+------------------------------------------------------------- rdsadmin | <insufficient privilege> rdsadmin | <insufficient privilege> gtest | SELECT c FROM sbtest27 WHERE id BETWEEN 392582 AND 392582+4 gtest | select usename, query from pg_stat_activity

NOW usename | query----------+---------------------------------------------- rdsadmin | <insufficient privilege> gtest | select usename, query from pg_stat_activity gtest | COMMIT rdsadmin | autovacuum: ANALYZE public.sbtest16

CloudWatch Metric

AWS Database BLOG

https://aws.amazon.com/blogs/database/



Thank you!

Questions?

Date post:	21-Apr-2017
Category:	Data & Analytics
Upload:	grant-mcalister
View:	37 times
Download:	6 times

Amazon RDS PostgreSQL: Enabling Innovation with Cloud Managed Databases

Data & Analytics